Memory Management when developing plug-ins

Question

I have done a search on this issue here and there are a lot of queries about memory issues and WordPress. The common recommended fix is to alter the WORDPRESS_MEMORY constant. I have a similar question and I would like to avoid the need to redefine this constant. Here is my question as posted at WordPress.org:

I am developing a plug-in that has a number of custom queries.

I am looking for some guidance on how best to approach a problem when the plug-in is operational.

After activation, I need a method for going through ALL posts, pages and custom post types (CPT) and adding 4 custom fields to each post, page and CPT.

Here is the code I am using for this:

$internal   = array( 'page', 'post' );
$cpt_args = array(
    'public' => true,
    '_builtin' => false
);
$custom = get_post_types( $cpt_args, 'objects' ); 

foreach ($custom as $name => $data) {
    $internal[] = $name;
}
$avail_posttypes = $internal;

$args = array(
    'post_type' => $avail_posttypes,
    'post_status'   => 'any',
    'numberposts'   => -1,
    'cache_results' => false,
    'no_found_rows' => true,
    'fields'    => 'ids',
);
$all_posts = get_posts($args);

foreach ($all_posts as $post) {
    if( !update_post_meta( $post->ID, '_cf_one', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_one', 'ok', true );
    }
    if( !update_post_meta( $post->ID, '_cf_two', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_two', 'ok', true );
    }
    if( !update_post_meta( $post->ID, '_cf_three', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_three', 'ok', true );
    }
    if( !update_post_meta( $post->ID, '_cf_four', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_four', 'ok', true );
    }
}

$img_args = array(
    'post_type' => 'attachment',
    'numberposts'   => -1,
    'cache_results' => false,
    'no_found_rows' => true,
    'fields'    => 'ids',
);
$img_posts = get_posts($img_args);

foreach ($img_posts as $post) {
    if( !update_post_meta( $post->ID, '_cf_one', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_one', 'ok', true );
    }
    if( !update_post_meta( $post->ID, '_cf_two', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_two', 'ok', true );
    }
    if( !update_post_meta( $post->ID, '_cf_three', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_three', 'ok', true );
    }
    if( !update_post_meta( $post->ID, '_cf_four', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_four', 'ok', true );
    }
}

I have tried to optimize my queries to not consume any unnecessary resources. But it seems to be running our of memory before it gets to the query to process the images.

The particular website I am testing this on has over 600 posts, pages and custom post types (combined).

I have altered my memory setting in wp-config.php to be 128M. But I would like to find a way to do this so that common users do not need to alter this value or anything in the php.ini file.

Is there more I could be doing to optimize this request? Is there any documentation on how WordPress manages memory in relationship to queries?

The Query Monitor plugin is also showing that I an consuming about 37M of memory when the plug-in is active with about 59 queries. I would like to reduce this considerably as the memory goes down to 20M when my plugin is deactivated.

I would like to be able to allocate memory as needed or restructure my queries in such a way as to not consume so much. Any suggestions would be greatly appreciated.

Answer 1

You could try to tackle this problem in blocks of posts instead of all of them at the same time, you are currently loading all the posts into the memory and then you query the database using these posts which isn't terribly efficient.

You could get the total amount of post, and then query them in blocks of 25 or something each. Then do your queries and start over again with the next 25.

If you need I could whip up some code to illustrate my answer.

EDIT:
I have created an example which hopefully solves your problem. I fixed the first block of queries so if this works you (or I) can apply it to the second block in the same manner.

Unfortunately I don't have a Wordpress install handy so there might be bugs. Also I´m not entirely sure if wp_count_posts and $avail_posttypes play nice together.

<?php 
$internal   = array( 'page', 'post' );
$cpt_args = array(
    'public' => true,
    '_builtin' => false
);
$custom = get_post_types( $cpt_args, 'objects' ); 

foreach ($custom as $name => $data) {
    $internal[] = $name;
}
$avail_posttypes = $internal;
// Changed the numberposts to 25 to only query 25 posts, and added an offset.
$args = array(
    'post_type' => $avail_posttypes,
    'post_status'   => 'any',
    'numberposts'   => 25,
    'offset'   => 0,
    'cache_results' => false,
    'no_found_rows' => true,
    'fields'    => 'ids',
);


$total_amount_of_posts = wp_count_posts($avail_posttypes);

// Check if there are more post waiting if so then query them.
while(args['offset']+args['numberposts']<$total_amount_of_posts)
{
    // Get the next 25 posts.
    $posts = get_posts($args);

foreach ($posts as $post) {
    if( !update_post_meta( $post->ID, '_cf_one', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_one', 'ok', true );
    }
    if( !update_post_meta( $post->ID, '_cf_two', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_two', 'ok', true );
    }
    if( !update_post_meta( $post->ID, '_cf_three', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_three', 'ok', true );
    }
    if( !update_post_meta( $post->ID, '_cf_four', 'ok', true ) ){
        add_post_meta( $post->ID, '_cf_four', 'ok', true );
    }
}
// Skip over the 25 we just queried.
args['offset'] += 25;

}