I'm processing a large JSON File (>1500 items). Im using JSONStreamParser to process one item at time. With each one, I am updating a post if the data has changed. The first thing I need to do is check to see if we already have that item in the database, so I am doing a lookup as shown below.

The problem is that each time the function below runs, it adds to the memory that is being used so that by the time we get through 500 items, the memory has grown by 50MB and then the whole process just quits.

Is there something I can do to release that memory? Is there a better way to do this?

function get_page_by_loclink($loclink) {
  $args = array(
    'post_type'   => 'cm-map-location',
    'meta_query'  => array(
        'key'     => 'ht_loclink',
        'value'   => $loclink,
        'compare' => '='
 $query = new WP_Query($args);
 if ($query->have_posts()) {
  $posts = $query->posts;
  return $posts[0]->ID;
 } else {
  return false;
  • 2
    Note that you're querying posts via their post meta in your code, which is extremely slow/inefficient. Do not store information in post meta if you intend to query for it, that's what taxonomies are for. – Tom J Nowell Apr 25 '18 at 18:54
  • I ended up creating a new table in the database where I can pair the two ids I need match up records. Then on subsequent updates I can just do a lookup on that table which is way less memory intensive. – PattyOK Apr 27 '18 at 0:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.