How do I split a large query with a semi-expensive function included into multiple smaller queries

Question

I have a count stored in post-meta called version-count. I just wrote a little plugin to delete the data stored in this post-meta using delete_post_meta_by_key.

I want my plugin to then go through each post and re-run my count-versions function, which saves the new count in post-meta.

The problem is that when I run this code, I get a 504 error. My query returns 8,000+ results.

$args= array(
    'post_type' => 'release',
    'posts_per_page' => -1,
    'tax_query' => array(
        array(
            'taxonomy' => 'release-format',
            'field' => 'slug',
            'terms' => 'album',
        ),
    ),
  );
  $newreleasequery= new WP_Query( $args );
  while ($newreleasequery->have_posts()) : $newreleasequery->the_post();
    $release_id=get_the_ID();
    get_album_version_count( $release_id );
    $count++;
  endwhile;
  echo 'All version counts have been reset. ', 'Count is ', $count;

There are too many posts in this query to run at once. How can I get it to piecemeal through the results and complete the function in chunks it can handle.

I'm guessing it has something to do with posts_per_page? Can I do something like this, and just keep writing more and more? It seems like there would be a faster/more efficient way to do this... Thanks all!

$args= array(
    'post_type' => 'release',
    'posts_per_page' => 250,
    'tax_query' => array(
        array(
            'taxonomy' => 'release-format',
            'field' => 'slug',
            'terms' => 'album',
        ),
    ),
  );
  $newreleasequery= new WP_Query( $args );
  $querycount= $newreleasequery->found_posts;
  //echo $querycount;
  
  while ($newreleasequery->have_posts()) : $newreleasequery->the_post();
    $release_id=get_the_ID();
    get_album_version_count( $release_id );
    
  endwhile;
  wp_reset_query();
  if($querycount >= 250){
    $args= array(
        'post_type' => 'release',
        'posts_per_page' => 250,
        'offset' => 250,
        'tax_query' => array(
            array(
                'taxonomy' => 'release-format',
                'field' => 'slug',
                'terms' => 'album',
            ),
        ),
    );
    $newreleasequery= new WP_Query( $args );
    $querycount= $newreleasequery->found_posts;
    //echo $querycount;
    
    while ($newreleasequery->have_posts()) : $newreleasequery->the_post();
        $release_id=get_the_ID();
        get_album_version_count( $release_id );
        
    endwhile;
    wp_reset_query();
  }if($querycount >= 500){
    $args= array(
        'post_type' => 'release',
        'posts_per_page' => 250,
        'offset' => 500,
        'tax_query' => array(
            array(
                'taxonomy' => 'release-format',
                'field' => 'slug',
                'terms' => 'album',
            ),
        ),
    );
    $newreleasequery= new WP_Query( $args );
    $querycount= $newreleasequery->found_posts;
    //echo $querycount;
    
    while ($newreleasequery->have_posts()) : $newreleasequery->the_post();
        $release_id=get_the_ID();
        get_album_version_count( $release_id );
        
    endwhile;
    wp_reset_query();
  }
  
  echo 'All version counts have been reset. ', 'Count is ', $querycount;

Answer 1

Yes, using 'posts_per_page' => -1 is a bad practice as it loads every matching post from the database, which can mean hundreds, thousands or millions of entries in the worst case.

And it looks like you only need post ID's in your code, not the whole post objects. This means you can slim down the query by adding 'fields' => 'ids' to the arguments to only retrieve the required post ID's.

In this case setting up the loop with have_posts() is also a bit unnecessary as the found posts can be accessed directly from the query object, e.g. $query->posts.

Instead of writing multiple queries and if statements, you can utilize a do-while loop with a paged query, which runs one or more times depending on how many pages of posts there are.

function wpse_410390_recount_versions_count() : void {
  $hasPosts = true;
  $paged = 0;
  
  do {
    $query = new WP_Query([
        'post_type' => 'release',
        'post_status' => ['publish', 'draft', 'future', 'private', 'pending'],
        'posts_per_page' => 100,
        'paged' => $paged,
        'tax_query' => array(
          array(
            'taxonomy' => 'release-format',
            'field' => 'slug',
            'terms' => 'album',
          ),
      ),
      'fields' => 'ids'
    ]);
  
    if ( $query->posts ) {
        foreach ( $query->posts as $post_id ) {
            some_function_to_update_versions_count($post_id);
        }
        $paged++;
    } else {
        $hasPosts = false;
    }
  } while ( $hasPosts );
} 

Depending on your setup you might be able to toss in to the query arguments 'update_post_meta_cache' => false and 'update_post_term_cache' => false to slim down the query a bit more.