How to solve suspected memory issue in custom WordPress loop?

Question

I have written a plugin that adds a "Where did you hear about us?" question to the WooCommerce checkout page and shows the results (as a simple horizontal bar chart) in a wp-admin page.

The plugin worked fine but has stopped working at some since I wrote it - seemingly, due to a memory issue.

In my wp-config.php file, I have... define('WP_MEMORY_LIMIT', '128M'); ...and I have also tried 256MB, 512MB and 1G - but not of these solve the problem.

phpinfo() reports a memory_limit of 1G - so that seems fine.

So here is the function for my plugin which is where PHP appears to give up:

function get_customer_sources() {

    $args = array(
        'post_type' => 'shop_order',
        'post_status' => 'publish',
        'posts_per_page' => -1,
        'tax_query' => array(
            array(
                'taxonomy' => 'shop_order_status',
                'field' => 'slug',
                'terms' => array('processing', 'completed', 'cancelled', 'failed', 'refunded')
            )
        )
    );

    //$custom_posts = new WP_Query($args);
    global $post; // required
    $custom_posts = get_posts($args);

    $all_sources = array(); //The return array
    echo "<h1>START</h1>";

    //while ($custom_posts->have_posts()) : $custom_posts->the_post();
    //if ( $custom_posts->have_posts() ) : while ( $custom_posts->have_posts() ) : $custom_posts->the_post();
    foreach($custom_posts as $post) : setup_postdata($post);


        //echo "<h2>A</h2>";

        $order_id = $post->ID;

        //echo "<h3>Order ID: $order_id</h3>";

        if ( get_post_meta($order_id, 'Where did you hear about us', true) ) {

            $source = get_post_meta($order_id, 'Where did you hear about us', true);

            //echo '<h4>Order ID: ' . $order_id . ': ' . $source . '</h4>';

            // Fix the "Vegan event" vs "Vegan Event" problem
            $source = ucfirst(strtolower($source));

            // Add the source to the return array
            if (array_key_exists($source, $all_sources)) {
                // Increment existing value
                $num = $all_sources[$source];
                $all_sources[$source] = $num + 1;
            }
            else {
                // Add value
                $all_sources[$source] = 1;
            }

            //echo '<h4>Num sources: ' . count($all_sources) . '</h4>';

        }

        //echo "<h2>B</h2>";

    //endwhile;
    //endwhile; endif;
    endforeach;

    echo "<h1>END</h1>";

    return $all_sources;
}`

I thought maybe there was some bad data that was messing things up, so I changed the...

if ( get_post_meta($order_id, 'Where did you hear about us', true) ) {

...line to...

if ( $order_id < 5000 && get_post_meta($order_id, 'Where did you hear about us', true) ) {

...and it worked fine.

Next, I changed that line to...

if ( $order_id >= 5000 && get_post_meta($order_id, 'Where did you hear about us', true) ) {

...and it still worked.

But without the 5000 condition, the script just bombs out. (I know this because <h1>END</h1> is not outputted.)

So what is wrong with my code and/or how can I solve this problem of the method not completing? For example, could it be written more efficiently in terms of memory? Or is there a better approach? (You'll see from the commented-out that I have already tried using WP_Query, but I get the same results.)

NB - There are approx. 10,000 items returned by the query.

Thanks in advance.

Answer 1

There are approx. 10,000 items returned by the query.

That's your problem right there. No matter what you do inside the loop, WordPress is still loading 10,000 post objects into memory.

Batch it up and sprinkle a little magic in your query arguments:

$args = array(
    'fields'         => 'ids', // MAGIC! Just get an array of id's, no objects committed to memory
    'posts_per_page' => 500,   // Batch size: big enough to reduce meta queries, small enough not to hammer memory
    'post_type'      => 'shop_order',
    'post_status'    => 'publish',
    'tax_query'      => array(
        array(
            'taxonomy' => 'shop_order_status',
            'field' => 'slug',
            'terms' => array( 'processing', 'completed', 'cancelled', 'failed', 'refunded' ),
        )
    )
);

$query = new WP_Query;
$paged = 1;
$count = 0;
$total = null;

do {
    $args['no_found_rows'] = isset( $total ); // No need to SQL_CALC_FOUND_ROWS on subsequent iterations
    $args['paged'] = $paged++;

    $post_ids = $query->query( $args );
    update_postmeta_cache( $post_ids ); // Get all meta for this group of posts

    if ( ! isset( $total ) )
        $total = $query->found_posts;

    $count += $query->post_count;

    foreach ( $post_ids as $post_id ) {
        $source = get_post_meta( $post_id, 'Where did you hear about us', true );

        // Do your stuff here



        // Wipe this post's meta from memory
        wp_cache_delete( $post_id, 'post_meta' );
    }

} while ( $count < $total );