Most efficient way to get posts with postmeta

Question

I need to get a bunch of posts with their metadata. Of course you can't get metadata with a standard posts query, so you generally have to do a get_post_custom() for each post.

I'm trying with one custom query, like this:

$results = $wpdb->get_results("
    SELECT  p.ID,
        p.post_title,
        pm1.meta_value AS first_field,
        pm2.meta_value AS second_field,
        pm3.meta_value AS third_field
    FROM    $wpdb->posts p LEFT JOIN $wpdb->postmeta pm1 ON (
            pm1.post_id = p.ID  AND
            pm1.meta_key    = 'first_field_key'
        ) LEFT JOIN $wpdb->postmeta pm2 ON (
            pm2.post_id = p.ID  AND
            pm2.meta_key    = 'second_field_key'
        ) LEFT JOIN $wpdb->postmeta pm3 ON (
            pm3.post_id = p.ID  AND
            pm3.meta_key    = 'third_field_key'
        )
    WHERE   post_status = 'publish'
");

Seems to work. It trips up if you use any of those meta fields in a way that allows multiple meta values for it on the same post. I can't think of a join to do that.

So, question 1: Is there a join, sub-query, or whatever, to bring in multiple-value meta fields?

But question 2: Is it worth it? How many postmeta table joins do I add before a 2-query approach becomes preferable? I could grab all post data in one query, then grab all relevant postmeta in another, and combine the meta with the post data in one resultset in PHP. Would that end up being quicker than an single ever-more-complex SQL query, if that's even possible?

I always think, "Give as much work as possible to the database." Not sure on this one!

Answer 1

Post meta information is automatically cached in memory for a standard WP_Query (and the main query), unless you specifically tell it not to do so by using the update_post_meta_cache parameter.

Therefore, you should not be writing your own queries for this.

How the meta caching works for normal queries:

If the update_post_meta_cache parameter to the WP_Query is not set to false, then after the posts are retrieved from the DB, then the update_post_caches function will be called, which in turn calls update_postmeta_cache.

The update_postmeta_cache function is a wrapper for update_meta_cache, and it essentially calls a simple SELECT with all the ID's of the posts retrieved. This will have it get all the postmeta, for all the posts in the query, and save that data in the object cache (using wp_cache_add).

When you do something like get_post_custom(), it's checking that object cache first. So it's not making extra queries to get the post meta at this point. If you've gotten the post in a WP_Query, then the meta is already in memory and it gets it straight from there.

Advantages here are many times greater than making a complex query, but the greatest advantage comes from using the object cache. If you use a persistent memory caching solution like XCache or memcached or APC or something like that, and have a plugin that can tie your object cache to it (W3 Total Cache, for example), then your whole object cache is stored in fast memory already. In which case, there's zero queries necessary to retrieve your data; it's already in memory. Persistent object caching is awesome in many respects.

In other words, your query is probably loads and loads slower than using a proper query and a simple persistent memory solution. Use the normal WP_Query. Save yourself some effort.

Additional: update_meta_cache() is smart, BTW. It won't retrieve meta information for posts that already have their meta information cached. It doesn't get the same meta twice, basically. Super efficient.

Additional additional: "Give as much work as possible to the database."... No, this is the web. Different rules apply. In general, you always want to give as little work as possible to the database, if it's feasible. Databases are slow or poorly configured (if you didn't configure it specifically, you can bet good money that this is true). Often they are shared among many sites, and overloaded to some degree. Usually you have more web servers than databases. In general, you want to just get the data you want out of the DB as fast and simply as possible, then do the sorting out of it using the web-server-side code. As a general principle, of course, different cases are all different.

Answer 2

I would recommend a pivot query. Using your example:

SELECT  p.ID,   
        p.post_title, 
        MAX(CASE WHEN wp_postmeta.meta_key = 'first_field' then wp_postmeta.meta_value ELSE NULL END) as first_field,
        MAX(CASE WHEN wp_postmeta.meta_key = 'second_field' then wp_postmeta.meta_value ELSE NULL END) as second_field,
        MAX(CASE WHEN wp_postmeta.meta_key = 'third_field' then wp_postmeta.meta_value ELSE NULL END) as third_field,

 FROM    wp_posts p LEFT JOIN wp_postmeta pm1 ON ( pm1.post_id = p.ID)                      
GROUP BY
   wp_posts.ID,wp_posts.post_title

Answer 3

I've come across a case where I want also want to quickly retrieve lots of posts with their associated meta information. I need to retrieve O(2000) posts.

I tried it using Otto's suggestion - running WP_Query::query for all posts, and then looping through and running get_post_custom for each post. This took, on average, about 3 seconds to complete.

I then tried Ethan's pivot query (though I didn't like having to manually ask for each meta_key I was interested in). I still had to loop through all retrieved posts to unserialize the meta_value. This took, on average, about 1.3 seconds to complete.

I then tried using the GROUP_CONCAT function, and found the best result. Here's the code:

global $wpdb;
$wpdb->query('SET SESSION group_concat_max_len = 10000'); // necessary to get more than 1024 characters in the GROUP_CONCAT columns below
$query = "
    SELECT p.*, 
    GROUP_CONCAT(pm.meta_key ORDER BY pm.meta_key DESC SEPARATOR '||') as meta_keys, 
    GROUP_CONCAT(pm.meta_value ORDER BY pm.meta_key DESC SEPARATOR '||') as meta_values 
    FROM $wpdb->posts p 
    LEFT JOIN $wpdb->postmeta pm on pm.post_id = p.ID 
    WHERE p.post_type = 'product' and p.post_status = 'publish' 
    GROUP BY p.ID
";

$products = $wpdb->get_results($query);

// massages the products to have a member ->meta with the unserialized values as expected
function massage($a){
    $a->meta = array_combine(explode('||',$product->meta_keys),array_map('maybe_unserialize',explode('||',$product->meta_values)));
    unset($a->meta_keys);
    unset($a->meta_values);
}

$products = array_map('massage',$products);

This took on average 0.7 seconds. That's about a quarter of the time of the WP get_post_custom() solution and about half of the pivot query solution.

Maybe this will be of interest to someone.

Answer 4

I ran into the multiple value meta fields problem as well. The problem is with WordPress itself. Look in wp-includes/meta.php. Look for this line:

$where[$k] = ' (' . $where[$k] . $wpdb->prepare( "CAST($alias.meta_value AS {$meta_type}) {$meta_compare} {$meta_compare_string})", $meta_value );

The problem is with the CAST statement. In a query for meta values, the $meta_type variable is set to CHAR. I don't know the details on how CASTing the value to CHAR affects the serialized string, but to fix it, you can remove the cast so the SQL looks like this:

$where[$k] = ' (' . $where[$k] . $wpdb->prepare( "$alias.meta_value {$meta_compare} {$meta_compare_string})", $meta_value );

Now, even though that works, you're mucking with the WordPress internals, so other things might break, and it's not a permanent fix, assuming you'll need to upgrade WordPress.

The way I've fixed it is to copy the SQL generated by WordPress for the meta query I want and then write some PHP to tack on extra AND statements for the meta_values I'm looking for and use $wpdb->get_results($sql) for the final output. Hacky, but it works.