meta_query with meta values as serialize arrays

Question

I'm working on a project in which I'm creating a custom post type and custom data entered via meta boxes associated with my custom post type. For whatever reason I decided to code the meta boxes in such a way that the inputs in each metabox are part of an array. For instance, I'm storing longitude and latitude:

<p> 
    <label for="latitude">Latitude:</label><br /> 
    <input type="text" id="latitude" name="coordinates[latitude]" class="full-width" value="" /> 
</p> 
<p>     
    <label for="longitude">Longitude:</label><br /> 
    <input type="text" id="longitude" name="coordinates[longitude]" class="full-width" value="" /> 
</p>

For whatever reason, I liked the idea of having a singular postmeta entry for each metabox. On the save_post hook, I save the data like so:

update_post_meta($post_id, '_coordinates', $_POST['coordinates']);

I did this because I have three metaboxes and I like just having 3 postmeta values for each post; however, I've now realized a potential issue with this. I may want to use WP_Query to only pull out certain posts based these meta values. For instance, I may want to get all posts that have latitude values above 50. If I had this data in the database individually, perhaps using the key latitude, I would do something like:

$args = array(
    'post_type' => 'my-post-type',
    'meta_query' => array(
        array(
            'key' => 'latitude',
            'value' => '50',
            'compare' => '>'
        )
    )
 );
$query = new WP_Query( $args );

Since I have the latitude as part of the _coordinates postmeta, this would not work.

So, my question is, is there a way to utilize meta_query to query a serialized array like I have in this scenario?

Answer 1

No, it is not possible, and could even be dangerous.

There is no reliable way to do this without encountering gotchas and pitfalls such as a search for 10 matching 100 or 210, or matching the wrong sub-value etc. Such solutions are unreliable and have performance and security issues.

Serialised data is an attack vector, and a major performance issue.

I'll cover:

• Fixing it and making it queryable
• Why Can't You Query Inside Serialised Data?
  • Why LIKE isn't a solution
• A Note on Storing Records/Entities/Objects as Serialized Objects in Meta
• Security and Serialized Objects
• What to do If I have a List of IDs?
• What to do If I Have an Array of Named items?
• Sidestepping The Problem Entirely
• Conclusion

But What About LIKE?

You might see some well meaning questions that suggest using LIKE to achieve this. Does this not solve the problem? This is not the solution, it is fools gold.

There are several major problems:

• false matches, searching for test with LIKE will also match test, testing, untested, and other values
• there's no way to constrict this to sub-keys for arrays with keys or objects
• it's not possible to do sorting
• it's extremely slow and expensive

LIKE will only work for specific limited situations that are unrealistic, and carries a heavy performance penalty.

Fixing It And Making It Queryable

I strongly recommend you unserialise your data and modify your save routine. Something similar to this should convert your data to the new format:

$args = [
    'post_type' => 'my-post-type',
    'meta_key' => '_coordinates',
    'posts_per_page' => -1,
];
$query = new WP_Query( $args );
if ( $query->have_posts() ) {
    while ( $query->have_posts() ) {
        $query->the_post();
        // get the data
        $c = get_post_meta( $post->ID, '_coordinates', true );

        // save it in the new format, separate post meta, taxonomy term etc
        add_post_meta( $post->ID, '_longitude', $c['longitude'] );
        add_post_meta( $post->ID, '_latitude', $c['latitude'] );

        // Optional: clean up the old post meta
        delete_post_meta( $post->ID, '_coordinates', $c );
    }
    wp_reset_postdata();
}

Then you'll be able to query as you want with individual keys

If you need to store multiple longitudes, and multiple latitudes, you can store multiple post meta with the same name. Simply use the third parameter of get_post_meta, and it will return them all as an array

Why Can't You Query Inside Serialised Data?

MySQL sees it as just a string, and can't break it apart into structured data. Breaking it apart into structured data is exactly what the code above does

You may be able to query for partial chunks of date, but this will be super unreliable, expensive, slow, and very fragile, with lots of edge cases. Serialised data isn't intended for SQL queries, and isn't formatted in a regular and constant way.

Aside from the costs of partial string searches, post meta queries are slow, and serialised data can change depending on things such as the length of contents, making searching incredibly expensive, if not impossible depending on the value you're searching for

A Note on Storing Records/Entities/Objects as Serialized Objects in Meta

You might want to store a transaction record in post meta, or some other kind of data structure in user meta, then run into the problem above.

The solution here is not to break it out into individual post meta, but to realise it should never have been meta to begin with, but a custom post type. For example, a log or record can be a custom post type, with the original post as a parent, or joined via a taxonomy term

Security and Serialized Objects

Storing serialized PHP objects via the serialize function can be dangerous, which is unfortunate as passing an object to WordPress will mean it gets serialised. This is because when the object is de-serialized, an object is created, and all its wake up methods and constructors get executed. This might not seem like a big deal until a user manages to sneak a carefully crafted input, leading to remote code execution when the data is read from the database and de-serialized by WordPress.

This can be avoided by using JSON instead, which also makes the queries easier, but it's much easier/faster to just store the data correctly and avoid structured serialized data to begin with.

What If I have a List of IDs?

You might be tempted to give WP an array, or to turn it into a comma separated list, but you don't have to!

Post meta keys are not unique, you can store the same key multiple times, e.g.:

$id = ...;
add_post_meta( $id, 'mylist', 1 );
add_post_meta( $id, 'mylist', 2 );
add_post_meta( $id, 'mylist', 3 );
add_post_meta( $id, 'mylist', 4 );
add_post_meta( $id, 'mylist', 5 );
add_post_meta( $id, 'mylist', 6 );

But how do I get the data back out? Have you ever noticed how calls to get_post_meta have a 3rd parameter that's always set to true? Set it to false:

$mylist = get_post_meta( $id, 'mylist', false );
foreach ( $mylist as $number ) {
    echo '<p>' . esc_html( $number ) . '</p>;
}

What If I Have an Array of Named items?

What if I wanted to store this data structure in a way that lets me query the fields?

{
  "foo": "bar",
  "fizz": "buzz"
  "parent": {
    "child": "value"
  }
}

That's easy, split it up with prefixes:

add_post_meta( $id, "tomsdata_foo", "bar" );
add_post_meta( $id, "tomsdata_fizz", "buzz" );
add_post_meta( $id, "tomsdata_parent_child", "value" );

And if you needed to loop over some of those values, use get_post_meta( $id ); to grab all post meta and loop over the keys, e.g.:

$all_meta = get_post_meta( $id );
$look_for = 'tomsdata_parent';
foreach ( $all_meta as $key => $value ) {
    if ( substr($string, 0, strlen($look_for)) !== $look_for ) {
        continue; // doesn't match, skip!
    }
    echo '<p>' . $key . ' = ' . $value . '</p>';
}

Which would output:

<p>tomsdata_parent_child = value</p>

Remember, when WP fetches a post it fetches all its post meta at the same time, so get_post_meta calls are super cheap and do not trigger extra database queries

Sidestepping The Problem Entirely

If you know you're going to need to search/query/filter on a sub-value, why not store an additional post meta with that value so you can search for it?

Conclusion

So you don't need to store structured data as a string in the database, and you shouldn't if you plan to search/query/filter on those values.

It might be possible to use a regular expression and a LIKE, but this is extremely unreliable, doesn't work for most types of data, and very, very slow and heavy on the database. You also can't perform math on the results like you could if they were separate values

Answer 2

I also run into this situation. Here what i did:

$args = array(
    'post_type' => 'my-post-type',
    'meta_query' => array(
        array(
            'key' => 'latitude',
            'value' => sprintf(':"%s";', $value),
            'compare' => 'LIKE'
        )
    )
);

Hope this help

Answer 3

You really are going to lose the ability to query your data in any efficient manner when serializing entries into the WP database.

The overall performance saving and gain you think you are achieving by serialization is not going to be noticeable to any major extent. You might obtain a slightly smaller database size but the cost of SQL transactions is going to be heavy if you ever query those fields and try to compare them in any useful, meaningful manner.

Instead, save serialization for data that you do not intend to query in that nature, but instead would only access in a passive fashion by the direct WP API call get_post_meta() - from that function you can unpack a serialized entry to access its array properties too.

In fact assigned the value of true as in;

$meta = get_post_meta( $post->ID, 'key', true );

Will return the data as an array, accessible for you to iterate over as per normal.

You can focus on other database/site optimizations such as caching, CSS and JS minification and using such services as a CDN if you require. To name but a few.... WordPress Codex is a good starting point to uncover more on that topic: HERE

Answer 4

I think there are 2 solutions that can try to solve the problem of results being stored as both String and Integers. However, it's important to say, as others pointed out, that it is not possible to guarantee the integrity of results stored as Integer, because as these values as stored as serialized arrays, the index and the values are stored exactly with the same pattern. Example:

array(37,87);

is stored as a serialized array, like this

a:2:{i:0;i:37;i:1;i:87;}

Note the i:0 as the first position of the array and i:37 as the first value. The pattern is the same. But let's go to the solutions

---

1) REGEXP Solution

This solution works for me regardless of the meta value being saved as string or number / id. However it uses REGEXP, which is not so fast as using LIKE

$args = array(
    'post_type' => 'my-post-type',
    'meta_query' => array(
        array(
            'key' => 'latitude',
            'value' => '\;i\:' . $value . '\;|\"' . $value . '\";',
            'compare' => 'REGEXP'
        )
    )
);

2) LIKE Solution

I'm not sure about the performance difference but this is a solution that uses LIKE and also works for both number and strings

 $args = array(
        'post_type' => 'my-post-type',
        'meta_query' => array(
            'relation' => 'OR',
            array(
                'key' => 'latitude',
                'value' => sprintf(':"%s";', $value),
                'compare' => 'LIKE'
            ),
            array(
                'key' => 'latitude',
                'value' => sprintf(';i:%d;', $value),
                'compare' => 'LIKE'
            )
        )
    );

Answer 5

I've just dealed with serialized fields and could query them. Not using the meta_query but using a SQL query.

global $wpdb; 

$search = serialize('latitude').serialize(50);

$query = $wpdb->prepare("SELECT `post_id`
FROM `wp_postmeta`
WHERE `post_id` IN (SELECT `ID` FROM `wp_posts` WHERE `post_type` = 'my-post-type')
AND `meta_key` = '_coordinates'
AND `meta_value` LIKE '%s'",'%'.$search.'%');

$ids = $wpdb->get_col($query);

$args = array(
    'post__in' => $ids
    'post_type' => 'team' //add the type because the default will be 'post'
);

$posts = get_posts($args);

The query first searches for post with the matching post_type so the amount of wp_postmeta records will be less to filter. Then i've added a where statement to reduce the rows further by filtering on meta_key

The IDs end up nicely in an array as needed for get_posts.

PS. MySQL v5.6 or higher is needed for good subquery performance

Answer 6

A simple hack to this situation is to use LIKE compare with serialised format (length with type and value itself) as below:

$id = 123; // id to check from serialised data.
//(assuming the numbers stored as string. In my case it was.)

$args = array(
    'post_type' => 'custom_post_type',
    'meta_query' => array(
        array(
            'key' => 'meta_key_to_check_from',
            'value' => 's:' . strlen($id) . ':"' . $id . '";',
            'compare' => 'LIKE'
        )
    )
);

$query = new WP_Query( $args );

If custom taxonomy to search:

$id = 123; // id to check into the serialised data.
//(assuming the numbers stored as string. In my case it was.)

$args = array(
    'hide_empty' => false, // also retrieve terms which are not used yet
    'meta_query' => array(
        array(
            'key' => 'meta_key_to_check_from',
            'value' => 's:' . strlen($id) . ':"' . $id . '";',
            'compare' => 'LIKE'
        )
    ),
    'taxonomy' => 'custom_taxonomy'
);

$terms = get_terms( $args );

Answer 7

This example really helped me. It's specifically for S2Members plugin (which serializes user metadata). But it allows you to query a portion of a serialized array within the meta_key.

It works by using the MySQL REGEXP function.

Here is the source

Here is the code that queries all users living in the US. I easily modified it to query one of my custom registration fields and had it working in no time.

  <?php
global $wpdb;
$users = $wpdb->get_results ("SELECT `user_id` as `ID` FROM `" . $wpdb->usermeta . 
          "` WHERE `meta_key` = '" . $wpdb->prefix . "s2member_custom_fields' AND 
           `meta_value` REGEXP '.*\"country_code\";s:[0-9]+:\"US\".*'");
if (is_array ($users) && count ($users) > 0)
    {
        foreach ($users as $user)
            {
                $user = /* Get full User object now. */ new WP_User ($user->ID);
                print_r($user); /* Get a full list of properties when/if debugging. */
            }
    }
?>

Answer 8

After reading a bunch of tips for running a WP_Query filtering by serialized arrays, here's how I finally did it: by creating an array of comma separated values using implode in conjunction with a $wpdb custom SQL query utilizing FIND_IN_SET to search the comma separated list for the requested value.

(this is similar to Tomas's answer, but its a bit less performance intensive for the SQL query)

1. In functions.php:

In your functions.php file (or wherever you're setting up the meta box) in the yourname_save_post() function use

update_post_meta($post->ID, 'checkboxArray', implode(",", $checkboxArray)); //adding the implode

to create the array containing comma separated values.

You'll also want to change your output variable in the yourname_post_meta() admin meta box construction function to

$checkboxArray = explode(",", get_post_custom($post->ID)["checkboxArray"][0]); //adding the explode

2. In the template PHP file:

Test: if you run a get_post_meta( $id ); you should see checkboxArray as an array containing your comma separated values instead of a serialized array.

Now, we build our custom SQL query using $wpdb.

global $wpdb;

$search = $post->ID;

$query = "SELECT * FROM wp_posts
          WHERE FIND_IN_SET( $search, (
              SELECT wp_postmeta.meta_value FROM wp_postmeta
              WHERE wp_postmeta.meta_key = 'blogLocations'
              AND wp_postmeta.post_id = wp_posts.ID )
          )
          AND ( wp_posts.post_type = 'post' )
          AND ( wp_posts.post_status = 'publish' );";

$posts = $wpdb->get_results($query);

foreach ($posts as $post) {
    //your post content here
}

Notice the FIND_IN_SET, that's where the magic happens.

Now... since I'm using SELECT * this returns all the post data and within the foreach you can echo out what you want from that (do a print_r($posts); if you don't know what's included. It doesn't set up "the loop" for you (I prefer it this way), but it can easily be modified to set up the loop if you prefer (take a look at setup_postdata($post); in the codex, you'll probably need to change SELECT * to select only post ID's and $wpdb->get_results to the correct $wpdb type -- see the codex for $wpdb also for information on that subject).

Whelp, it took a bit of effort, but since wp_query doesn't support doing 'compare' => 'IN' serialized or comma separated values this shim is your best option!

Hope this helps someone.

Answer 9

If you use the like comparison operator in your meta query, it should work fine to look inside a serialized array.

$wp_user_search = new WP_User_Query(array(
    'meta_query' => array(
        array(
            'key'     => 'wp_capabilities',
            'value'   => 'subscriber',
            'compare' => 'not like'
            )
        )
    )
);

results in:

[query_where] => WHERE 1=1 AND (
  ( wp_usermeta.meta_key = 'wp_capabilities' 
  AND CAST(wp_usermeta.meta_value AS CHAR) NOT LIKE '%subscriber%' )

Answer 10

If my meta data is array type, i'm use this method for query by meta:

$args = array(
    'post_type' => 'fotobank',
    'posts_per_page' => -1,
    'meta_query' => array(
            array(
                   'key' => 'collections',
                   'value' => ':"'.$post->ID.'";',
                   'compare' => 'LIKE'
            )
     )
);
$fotos = new WP_Query($args);

Answer 11

I got curious about the answers above, where the meta_query targeted the key latitude instead of _coordinates. Had to go and test if it really was possible in meta queries to target a specific key inside a serialized array. :)

That obviously wasn't the case.

So, note that the correct key to target is _coordinates instead of latitude.

$args = array(
     'post_type' => 'my-post-type',
     'meta_query' => array(
         array(
             'key' => '_coordinates',
             'value' => sprintf(':"%s";', $value),
             'compare' => 'LIKE'
         )
     )
 );

NOTES:

1. This approach makes it only possible to target exact matches. So things like all latitudes greater than 50 are not possible.

2. To include substring matches, one could use 'value' => sprintf(':"%%%s%%";', $value),. (haven't tested)

Answer 12

You can enclose the value with % to query the serialized value:

$args = array(
    'post_type' => 'my-post-type',
    'meta_query' => array(
        array(
            'key' => 'latitude',
            'value' => '%' . 50 . '%',
            'compare' => '>'
        )
    )
 );
$query = new WP_Query( $args );

Answer 13

I have the same question. Maybe you need the 'type' parameter? Check out this related question: Custom Field Query - Meta Value is Array

Perhaps try:

    $args = array(
    'post_type' => 'my-post-type',
    'meta_query' => array(
        array(
            'key' => 'latitude',
            'value' => '50',
            'compare' => '>',
            'type' => 'numeric'
        )
    )
    );

Answer 14

I ran into something similar while using the Magic Fields plugin. This might do the trick

$values_serialized = serialize(array('50'));
$args = array(
    'post_type' => 'my-post-type',
    'meta_query' => array(
        array(
            'key' => 'latitude',
            'value' => $values_serialized,
            'compare' => '>'
        )
    )
);