Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I need to get a bunch of posts with their metadata. Of course you can't get metadata with a standard posts query, so you generally have to do a get_post_custom() for each post.

I'm trying with one custom query, like this:

$results = $wpdb->get_results("
    SELECT  p.ID,
        p.post_title,
        pm1.meta_value AS first_field,
        pm2.meta_value AS second_field,
        pm3.meta_value AS third_field
    FROM    $wpdb->posts p LEFT JOIN $wpdb->postmeta pm1 ON (
            pm1.post_id = p.ID  AND
            pm1.meta_key    = 'first_field_key'
        ) LEFT JOIN $wpdb->postmeta pm2 ON (
            pm2.post_id = p.ID  AND
            pm2.meta_key    = 'second_field_key'
        ) LEFT JOIN $wpdb->postmeta pm3 ON (
            pm3.post_id = p.ID  AND
            pm3.meta_key    = 'third_field_key'
        )
    WHERE   post_status = 'publish'
");

Seems to work. It trips up if you use any of those meta fields in a way that allows multiple meta values for it on the same post. I can't think of a join to do that.

So, question 1: Is there a join, sub-query, or whatever, to bring in multiple-value meta fields?

But question 2: Is it worth it? How many postmeta table joins do I add before a 2-query approach becomes preferable? I could grab all post data in one query, then grab all relevant postmeta in another, and combine the meta with the post data in one resultset in PHP. Would that end up being quicker than an single ever-more-complex SQL query, if that's even possible?

I always think, "Give as much work as possible to the database." Not sure on this one!

share|improve this question
    
I'm not sure if you even want to do the joins. the combination of get_posts() and get_post_meta() gives you the same data back. In fact, it's less efficient using the joins since you might be retrieving data you won't use later. –  rexposadas Jan 10 '12 at 23:00
1  
Isn't post meta data cached automatically anyway? –  Manny Fleurmond Jan 11 '12 at 1:13
    
@rxn, if I have several hundred posts coming back (they're a custom post type), surely it's quite a heavy DB load to get_posts(), then get_post_meta() for every one of those? @MannyFleurmond, it's hard to find hard info on WP's built-in caching, but AFAIK it would cache stuff per request. The call to the server to grab this data is an AJAX call, and I don't think anything else will be grabbing stuff before it. –  Steve Taylor Jan 11 '12 at 13:01
    
Actually, I'm going for multiple queries and caching the results. It turns out we not only need post meta, including fields that have multiple values, we also need data on users connected to the posts via meta fields (two sets of these), plus user meta on them. Pure SQL is definitely out of the window! –  Steve Taylor Jan 11 '12 at 14:20

5 Answers 5

using the solution form trevor and modifying it to work with nested SQL. This is not tested.

global $wpdb;
$query = "
    SELECT p.*, (select pm.* From $wpdb->postmeta AS pm WHERE pm.post_id = p.ID)
    FROM $wpdb->posts p 
    WHERE p.post_type = 'product' and p.post_status = 'publish' 
";
$products = $wpdb->get_results($query);
share|improve this answer

I've come across a case where I want also want to quickly retrieve lots of posts with their associated meta information. I need to retrieve O(2000) posts.

I tried it using Otto's suggestion - running WP_Query::query for all posts, and then looping through and running get_post_custom for each post. This took, on average, about 3 seconds to complete.

I then tried Ethan's pivot query (though I didn't like having to manually ask for each meta_key I was interested in). I still had to loop through all retrieved posts to unserialize the meta_value. This took, on average, about 1.3 seconds to complete.

I then tried using the GROUP_CONCAT function, and found the best result. Here's the code:

global $wpdb;
$wpdb->query('SET SESSION group_concat_max_len = 10000'); // necessary to get more than 1024 characters in the GROUP_CONCAT columns below
$query = "
    SELECT p.*, 
    GROUP_CONCAT(pm.meta_key ORDER BY pm.meta_key DESC SEPARATOR '||') as meta_keys, 
    GROUP_CONCAT(pm.meta_value ORDER BY pm.meta_key DESC SEPARATOR '||') as meta_values 
    FROM $wpdb->posts p 
    LEFT JOIN $wpdb->postmeta pm on pm.post_id = p.ID 
    WHERE p.post_type = 'product' and p.post_status = 'publish' 
    GROUP BY p.ID
";

$products = $wpdb->get_results($query);

// massages the products to have a member ->meta with the unserialized values as expected
function massage($a){
    $a->meta = array_combine(explode('||',$product->meta_keys),array_map('maybe_unserialize',explode('||',$product->meta_values)));
    unset($a->meta_keys);
    unset($a->meta_values);
}

$products = array_map('massage',$products);

This took on average 0.7 seconds. That's about a quarter of the time of the WP get_post_custom() solution and about half of the pivot query solution.

Maybe this will be of interest to someone.

share|improve this answer
    
I would be interested in what results you get with a persistent object cache solution. The object cache will sometimes be slower for the base case, depending on your database and configuration, but real world results with a majority of hosts will give widely varying results. Memory based caching is ridiculously fast. –  Otto Oct 13 '12 at 3:07
    
Hey @Otto. Regardless of which method I use to get the data, I definitely want to cache the result. I've tried using the transient API to do it, but I'm hitting memory problems. The serialized string for my 2000 objects clocks at ~8M and set_transient() fails (memory exhausted). Also, have to change the max_allowed_packet MySQL setting. I'll look into caching it to file, but I'm not sure yet of the the performance there. Is there a way to cache to memory that persists across requests? –  Trevor Mills Oct 29 '12 at 20:52
    
...reading about xCache now. –  Trevor Mills Oct 29 '12 at 20:58
    
Yes, if you have a persistent memory cache (XCache, memcached, APC, etc), and use an object caching plugin (W3 Total Cache supports many types of memory caches), then it stores all the object cache in memory, giving you a many-fold speedup of pretty much everything. –  Otto Nov 10 '12 at 17:09
    
This is definitely creative. I pretty interesting solution... thanks. –  Jake Dec 14 '13 at 6:08

I ran into the multiple value meta fields problem as well. The problem is with WordPress itself. Look in wp-includes/meta.php. Look for this line:

$where[$k] = ' (' . $where[$k] . $wpdb->prepare( "CAST($alias.meta_value AS {$meta_type}) {$meta_compare} {$meta_compare_string})", $meta_value );

The problem is with the CAST statement. In a query for meta values, the $meta_type variable is set to CHAR. I don't know the details on how CASTing the value to CHAR affects the serialized string, but to fix it, you can remove the cast so the SQL looks like this:

$where[$k] = ' (' . $where[$k] . $wpdb->prepare( "$alias.meta_value {$meta_compare} {$meta_compare_string})", $meta_value );

Now, even though that works, you're mucking with the WordPress internals, so other things might break, and it's not a permanent fix, assuming you'll need to upgrade WordPress.

The way I've fixed it is to copy the SQL generated by WordPress for the meta query I want and then write some PHP to tack on extra AND statements for the meta_values I'm looking for and use $wpdb->get_results($sql) for the final output. Hacky, but it works.

share|improve this answer
    
I've not tried it, but leveraging the get_meta_sql filter that follows this line would of course be preferable to hacking core code. –  Steve Taylor Apr 25 '12 at 10:29

Post meta information is automatically cached in memory for a standard WP_Query (and the main query), unless you specifically tell it not to do so by using the update_post_meta_cache parameter.

Therefore, you should not be writing your own queries for this.

How the meta caching works for normal queries:

If the update_post_meta_cache parameter to the WP_Query is not set to false, then after the posts are retrieved from the DB, then the update_post_caches function will be called, which in turn calls update_postmeta_cache.

The update_postmeta_cache function is a wrapper for update_meta_cache, and it essentially calls a simple SELECT with all the ID's of the posts retrieved. This will have it get all the postmeta, for all the posts in the query, and save that data in the object cache (using wp_cache_add).

When you do something like get_post_custom(), it's checking that object cache first. So it's not making extra queries to get the post meta at this point. If you've gotten the post in a WP_Query, then the meta is already in memory and it gets it straight from there.

Advantages here are many times greater than making a complex query, but the greatest advantage comes from using the object cache. If you use a persistent memory caching solution like XCache or memcached or APC or something like that, and have a plugin that can tie your object cache to it (W3 Total Cache, for example), then your whole object cache is stored in fast memory already. In which case, there's zero queries necessary to retrieve your data; it's already in memory. Persistent object caching is awesome in many respects.

In other words, your query is probably loads and loads slower than using a proper query and a simple persistent memory solution. Use the normal WP_Query. Save yourself some effort.

Additional: update_meta_cache() is smart, BTW. It won't retrieve meta information for posts that already have their meta information cached. It doesn't get the same meta twice, basically. Super efficient.

Additional additional: "Give as much work as possible to the database."... No, this is the web. Different rules apply. In general, you always want to give as little work as possible to the database, if it's feasible. Databases are slow or poorly configured (if you didn't configure it specifically, you can bet good money that this is true). Often they are shared among many sites, and overloaded to some degree. Usually you have more web servers than databases. In general, you want to just get the data you want out of the DB as fast and simply as possible, then do the sorting out of it using the web-server-side code. As a general principle, of course, different cases are all different.

share|improve this answer
    
Otto, you are the best ) –  Alexey Oct 8 '13 at 21:36
    
One of the coolest things I have read in a while. Very informative... Thanks Otto. –  Jake Dec 14 '13 at 0:39

I would recommend a pivot query. Using your example:

SELECT  p.ID,   
        p.post_title, 
        MAX(CASE WHEN wp_postmeta.meta_key = 'first_field' then wp_postmeta.meta_value ELSE NULL END) as first_field,
        MAX(CASE WHEN wp_postmeta.meta_key = 'second_field' then wp_postmeta.meta_value ELSE NULL END) as second_field,
        MAX(CASE WHEN wp_postmeta.meta_key = 'third_field' then wp_postmeta.meta_value ELSE NULL END) as third_field,

 FROM    wp_posts p LEFT JOIN wp_postmeta pm1 ON ( pm1.post_id = p.ID)                      
GROUP BY
   wp_posts.ID,wp_posts.post_title
share|improve this answer
    
This answer should be marked correct. –  Luke Mar 13 '12 at 6:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.