The API you offer in a plugin or a theme depends on the logic of that specific code. There is probably no guide that applies to all situations.
I am a contributor for multiple plugins with APIs, and what I have learned so far is:
Do not offer an API until you really know how people use your code.
Release the first two or three versions without any API. No custom actions or filters, no public methods or functions (and never any global variables) if possible. Wait for requests from your users, but don’t add the code until you know your inner code structure will work in the long run.
Maintaining backwards compatibility of an API is hard. It might prevent necessary improvements on other places. Think about all the global variables WordPress cannot remove nowadays. That’s a bad API, and we are stuck with it for many years, because people use it already.
Consider separating your API from the rest of the code (see previous link for an idea).
Your API should be useful not only for third party developers, but for you too. Do not add restriction for yourself if you don’t have to.Eat your own dog food.
If you offer custom hooks, use them in your code. This will give other developers useful examples, and you can see possible flaws soon enough.
If the WordPress core would use the so-called Settings API internally, we wouldn’t have this mess today. Maybe.Lead by example.
Use the good parts of the WordPress core API in your plugin. Avoid anonymous objects, constants, global variables and any kind of unpredictable code.Make sure you are using a consistent naming scheme (not such a mess), and prefix everything.
Write the documentation first. Release a new (part of an) API later.
Create useful examples for everything. You will be amazed to see how many holes and redundancies you will find.Avoid callback hell.
Offer specific tools to debug your API when things don’t work as they should (including not minified scripts and stylesheets). I have written an example for how to debug AJAX, just to illustrate how creative you can be here. Again, these tools should be explained in your documentation before you release them.An alternative to WordPress’ callback paradigma could be the Observer pattern. This would raise the barrier for third-party developers, but it could result in better code on both sides.