After a few months of doing this, I believe that my original question is very well founded and while this technique is surely not new, here's what I'm doing to greatly optimize a lot of things in my system:
(It's more philosophical, but I'll provide code)
I'm not going to discuss the, hopefully, understood fact that you shouldn't transient items / queries that are going to impact your work-flow, nor should you transient everything, because that overloads the database, especially with a lot of big queries - an example would be, on a not-so-good host, transienting the posts of each user (assuming you have a big community) in throves is a big no-no. You might have inactive users that have 10-15 posts each, but no one goes to their page, so their data will just stay idle in the database, just waiting for someone to access it to reset. (This is solved by just creating the transient on that user's page, which, hopefully, you're already doing!)
Regardless, that's exactly how we must do it: Instead of thinking of transients in terms of expire-on-time, we should approach them from a expire-on-trigger way.
So, instead of:
set_transient( 'name', 'value', 3600 );
We should simply set the transient without a time and let a "cleaner object" handle it. Enter architectural choices. We must look at what exactly that transient saves / in what manner and decide when exactly it is best to delete it. So, let's assume we're dealing with an user's posts and that we won't re-update his transient on a timer. What we can do is hook to save_post
and just re-create that transient every time the user posts a post. This way, our transient will only ever update itself when the user himself is creating a new post:
add_action( 'save_post', function( $post_id, $post, $update ) {
$author = get_post_field( 'post_author', $post_id );
/**
* But if he is, then we'll delete his posts' transient.
*/
delete_transient( 'user_posts_' . $author->ID );
});
What we achieved here is never deleting a transient, unless it needs to be deleted.
Now, that transient will be re-created when somebody hits that user's page and not be on a timer.
The only problem is that the database now grows larger, so we need a garbage collection mechanism that doesn't keep transients that aren't "popular" in the database, so we don't clutter it.
I've successfully applied this to transients related to the media library images, where I would need a transient, but I'd also like a trigger to update it.
Finishing this, the more I look at this, the more I think it should be called "knowing when you need to update a transient".
So far, it has solved way more issues for me than it caused but I agree that it's uncertain whether or not this would scale well. At the same time the issue of needing to scale is a great issue to have and that, if you code the system this way, you only need to disconnect these deletion mechanisms for it to work in the traditional way.
So, exactly what has this helped me achieve? What are the benefits?
Transients without actually feeling like they're transients. Now, whenever you do an action, you see its result, where-as with the normal way, you'd need to wait for a time to see the effect (after the transient expired). Just refresh and the new data is there, no need to wait, no need to lose on the benefits.
In fact, it somehow makes more sense to delete transients on a certain trigger rather than on a timer.