Uploading images to Media Library fails with Memory Exhausted

Question

I am trying to upload many hundreds of images daily from a folder on the server to the media library using the following script that is scheduled via CRON:

<?php
require_once('../../../../public/wordpress/wp-load.php');
require_once('../../../../public/wordpress/wp-admin/includes/image.php');

function importImage($imagePath, $postId)
{
    $succeededFileCount = 0;
    $failedFileCount = 0;
    $files = scandir($imagePath);

    foreach ($files as $file) {
        if (in_array($file, ['.', '..'])) {
            continue;
        }

        $newPath = $imagePath . "/" . $file;
        $filePath = realpath($newPath);

        if (is_dir($newPath) && $item != '.' && $item != '..' && $item != 'failed_files') {
            importImage($newPath, $postId);
        } elseif ($item != '.' && $item != '..' && $item != 'failed_files') {
            $filename = basename($file);
            $uploadFile = wp_upload_bits($filename, null, file_get_contents($filePath));
            $wp_upload_dir = wp_upload_dir();

            if (! $uploadFile['error']) {
                $fileType = wp_check_filetype($filename, null);
                $attachment = [
                    'guid' => $wp_upload_dir['url'] . '/' . basename( $filename ),
                    'post_mime_type' => $fileType['type'],
                    'post_parent' => $postId,
                    'post_title' => preg_replace('/\.[^.]+$/', '', $filename),
                    'post_content' => '',
                    'post_status' => 'inherit'
                ];
                $attachmentId = wp_insert_attachment($attachment, $uploadFile['file'], $postId);

                if (! is_wp_error($attachmentId)) {
                    $attachmentData = wp_generate_attachment_metadata($attachmentId, $uploadFile['file']);
                    wp_update_attachment_metadata($attachmentId, $attachmentData);
                }
            } else {
                echo '<span style="color: red; font-weight: bold;">Error: ' . $uploadFile['error'] . '</span>';
            }


            if ($attachmentId > 0) {
                $succeededFileCount++;
                echo '<span style="color: green; font-weight: normal;">File import succeeded: ' . $filePath . "</span><br />";

                if (! unlink($filePath)) {
                    echo '<span style="color: red; font-weight: bold;">Unable to delete file ' . $filePath . " after import.</span><br />";
                }

                $page = get_post($postId);

                if ($page->post_content) {
                    $content = $page->post_content;
                    $start = strpos($content, "[gallery ") + strlen("[gallery ");
                    $end = strpos(substr($content, $start), "]");
                    $shortcode = substr($content, $start, $end);
                    $attrs = shortcode_parse_atts($shortcode);
                    $attrs["ids"] .= "," . $attachmentId;
                    $tempIds = explode(",", $attrs["ids"]);
                    $tempIds = array_filter($tempIds);
                    rsort($tempIds);
                    $attrs["ids"] = implode(",", $tempIds);
                    $shortcode = "";

                    foreach ($attrs as $key => $value) {
                        if (strlen($shortcode) > 0) {
                            $shortcode .= " ";
                        }

                        $shortcode .= $key . "=\"" . $value . "\"";
                    }

                    $newContent = substr($content, 0, $start);
                    $newContent .= $shortcode;
                    $newContent .= substr($content, $start + $end, strlen($content));
                    $page->post_content = $newContent;
                    wp_update_post($page);
                }
            }
        }
    }

    echo $succeededFileCount . " files uploaded and imported successfully. <br />";
    echo $failedFileCount . " files failed to uploaded or import successfully.";
}

get_header();

if (get_option('rmm_image_importer_key') != urldecode($_GET['key'])) {
    echo '<div id="message" class="error">';
    echo "<p><strong>Incorrect authentication key: you are not allowed to import images into this site.</strong></p></div>";
} else {
    echo '<br /><br />';
    $mtime = microtime();
    $mtime = explode(" ", $mtime);
    $mtime = $mtime[1] + $mtime[0];
    $starttime = $mtime;
    $dataset = get_option('rmm_image_importer_settings');
    if (is_array($dataset)) {
        foreach ($dataset as $data) {
            if (isset($data['folder'])
                || isset($data['page'])) {
?>
    <h2>Import from folder: <?php echo $data['folder']; ?></h2>
    <p>
<?php
                importImage(realpath(str_replace('//', '/', ABSPATH . '../../' . $data['folder'])), $data['page']); ?>
    </p>
<?php
            }
        }
    }
    $mtime = microtime();
    $mtime = explode(" ", $mtime);
    $mtime = $mtime[1] + $mtime[0];
    $endtime = $mtime;
    $totaltime = ($endtime - $starttime);
    echo 'Files imported to media library over ' . $totaltime . ' seconds.<br /><br />';
}

get_footer();

The problem is that no matter what I do, the script fails after just two images with this error:

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 28672 bytes) in /home/forge/morselandcompany.com/public/wordpress/wp-includes/wp-db.php on line 1841

I have the memory limit set to 1024M in wordpress, as well as PHP. I don't see why this script should need more than 128M, really. How can this script be optimized to work properly? (Average image size is 800kB.)

Some initial debugging with BlackFire.io suggests the following memory hogs: - wpdb->query: 3.2MB, called 31 times - mysqli_fetch_object: 1.89MB, called 595 times - run_init() in wp-settings.php: 5.4MB, called once In total blackfire suggests that over 8MB is required to run this script!

I have also tested with all plugins disabled, and that ended with the same result.

I am running on - PHP 7.1
- Ubuntu 16.04
- DigitalOcean VPS (1 CPU, 1GB RAM)
- Wordpress 4.8
- NGINX 1.11.5

Thanks for any help!

Update: in the interest of completeness so that others may utilize the solution for memory leaks associated with get_post and wp_update_post, I have posted the finalized code that solved the problem above. As you can see, the solution was to write my own queries using $wpdb instead of relying on the two WP methods causing the memory leaks:

<?php

require_once('../../../../public/wordpress/wp-load.php');
require_once(ABSPATH . 'wp-admin/includes/media.php');
require_once(ABSPATH . 'wp-admin/includes/file.php');
require_once(ABSPATH . 'wp-admin/includes/image.php');

function importImage($imagePath, $postId)
{
    $succeededFileCount = 0;
    $failedFileCount = 0;
    $files = scandir($imagePath);

    foreach ($files as $file) {
        if (in_array($file, ['.', '..'])) {
            continue;
        }

        $newPath = $imagePath . "/" . $file;
        $filePath = realpath($newPath);

        if (is_dir($newPath) && $file != 'failed_files') {
            importImage($newPath, $postId);
        } elseif ($file != 'failed_files') {
            $webPath = str_replace($_SERVER['DOCUMENT_ROOT'], '', $imagePath);
            $imageUrl = str_replace('/wordpress', '', get_site_url(null, "{$webPath}/" . urlencode($file)));
            $imageUrl = str_replace(':8000', '', $imageUrl);
            $attachmentId = media_sideload_image($imageUrl, 0, '', 'id');

            if ($attachmentId > 0) {
                $succeededFileCount++;
                echo '<span style="color: green; font-weight: normal;">File import succeeded: ' . $filePath . "</span><br />";

                if (! unlink($filePath)) {
                    echo '<span style="color: red; font-weight: bold;">Unable to delete file ' . $filePath . " after import.</span><br />";
                }

                global $wpdb;
                $page = $wpdb->get_results("SELECT * FROM wp_posts WHERE ID = {$postId}")[0];

                if (is_array($page)) {
                    $page = $page[0];
                }

                if ($page->post_content) {
                    $content = $page->post_content;
                    $start = strpos($content, "[gallery ") + strlen("[gallery ");
                    $end = strpos(substr($content, $start), "]");
                    $shortcode = substr($content, $start, $end);
                    $attrs = shortcode_parse_atts($shortcode);
                    $attrs["ids"] .= "," . $attachmentId;
                    $tempIds = explode(",", $attrs["ids"]);
                    $tempIds = array_filter($tempIds);
                    rsort($tempIds);
                    $attrs["ids"] = implode(",", $tempIds);
                    $shortcode = "";

                    foreach ($attrs as $key => $value) {
                        if (strlen($shortcode) > 0) {
                            $shortcode .= " ";
                        }

                        $shortcode .= $key . "=\"" . $value . "\"";
                    }

                    $newContent = substr($content, 0, $start);
                    $newContent .= $shortcode;
                    $newContent .= substr($content, $start + $end, strlen($content));
                    $wpdb->update(
                        'post_content',
                        ['post_content' => $newContent],
                        ['ID' => $postId]
                    );
                }
            }
        }
    }

    echo $succeededFileCount . " files uploaded and imported successfully. <br />";
    echo $failedFileCount . " files failed to uploaded or import successfully.";
}

get_header();

if (get_option('rmm_image_importer_key') != urldecode($_GET['key'])) {
    echo '<div id="message" class="error">';
    echo "<p><strong>Incorrect authentication key: you are not allowed to import images into this site.</strong></p></div>";
} else {
    echo '<br /><br />';
    $mtime = microtime();
    $mtime = explode(" ", $mtime);
    $mtime = $mtime[1] + $mtime[0];
    $starttime = $mtime;
    $dataset = get_option('rmm_image_importer_settings');

    if (is_array($dataset)) {
        foreach ($dataset as $data) {
            if (isset($data['folder'])
                || isset($data['page'])) {
?>
    <h2>Import from folder: <?php echo $data['folder']; ?></h2>
    <p>
<?php
                importImage(realpath(str_replace('//', '/', ABSPATH . '../../' . $data['folder'])), $data['page']); ?>
    </p>
<?php
            }
        }
    }
    $mtime = microtime();
    $mtime = explode(" ", $mtime);
    $mtime = $mtime[1] + $mtime[0];
    $endtime = $mtime;
    $totaltime = ($endtime - $starttime);
    echo 'Files imported to media library over ' . $totaltime . ' seconds.<br /><br />';
}

get_footer();

Answer 1

A few things

• Use media_handle_sideload so that WordPress moves the files to the right location and validates them for you, creates the attachment post etc, none of this manual stuff
• Don't run this once and expect it to do everything. You're just going to run into the same problem but further into the import. If you give it infinite memory you'll have a time limit execution problem where the script simply runs out of time
• Process 5 files at a time, and repeatedly call run it until nothing is left to process
• Use a WP CLI command, don't bootstrap WordPress and call it from the GUI. Call it from Cron directly and skip the pinging a URL business. CLI commands get unlimited time to do their job, and you can't call them from the browser. The GET variable with the key becomes completely unnecessary.
• Escape escape escape, you're echoing out these arrays and values, assuming what they contain is safe, but what if I snuck a script tag in there? Unable to delete file <script>...</script> after import.. Biggest security step you can take that makes the most difference yet the least used

A Prototype WP CLI command

Here is a simple WP CLI command that should do the trick. I've not tested it but all the important parts are there, I trust you're not a complete novice when it comes to PHP and can tighten any loose screws or minor mistakes, no additional API knowledge necessary.

You'll want to only include it when in a WP CLI context, for example:

if ( defined( 'WP_CLI' ) && WP_CLI ) {
    require_once dirname( __FILE__ ) . '/inc/class-plugin-cli-command.php';
}

Modify accordingly, dumping the command in functions.php of a theme and expecting it all to work will cause errors as WP CLI classes are only loaded on the command line, never when handling a browser request.

Usage:

wp mbimport run

Class:

<?php
/**
 * Implements image importer command.
 */
class MBronner_Import_Images extends WP_CLI_Command {

    /**
     * Runs the import script and imports several images
     *
     * ## EXAMPLES
     *
     *     wp mbimport run
     *
     * @when after_wp_load
     */
    function run( $args, $assoc_args ) {
        if ( !function_exists('media_handle_upload') ) {
            require_once(ABSPATH . "wp-admin" . '/includes/image.php');
            require_once(ABSPATH . "wp-admin" . '/includes/file.php');
            require_once(ABSPATH . "wp-admin" . '/includes/media.php');
        }

        // Set the directory
        $dir = ABSPATH .'/wpse';
        // Define the file type
        $images = glob( $dir . "*.jpg" );
        if ( empty( $images ) {
            WP_CLI::success( 'no images to import' );
            exit;
        }
        // Run a loop and transfer every file to media library
        // $count = 0;
        foreach ( $images as $image ) {
            $file_array = array();
            $file_array['name'] = $image;
            $file_array['tmp_name'] = $image;

            $id = media_handle_sideload( $file_array, 0 );
            if ( is_wp_error( $id ) ) {
                WP_CLI::error( "failed to sideload ".$image );
                exit;
            }

            // only do 5 at a time, dont worry we can run this
            // several times till they're all done
            $count++;
            if ( $count === 5 ) {
                break; 
            }
        }
        WP_CLI::success( "import ran" );
    }
}

WP_CLI::add_command( 'mbimport', 'MBronner_Import_Images' );

Call repeatedly from a real cron job. If you can't, then either use WP Cron, or have a hook on admin_init that checks for a GET variable. Use the code inside the run command with some modifications.

When WP CLI Isn't An Option

Using a standalone PHP file that bootstraps WP is a security risk and a great target for attackers if they want to exhaust your server resources ( or trigger duplication issues by hitting the URL multiple times all at once ).

For example:

// example.com/?mbimport=true
add_action( 'init', function() {
    if ( $_GET['action'] !== 'mbimport' ) {
        return;
    }
    if ( $_GET['key'] !== get_option('key thing' ) ) {
        return;
    }
    // the code from the run function in the CLI command, but with the WP_CLI::success bits swapped out
    // ...
    exit;
}

Repeated Calling

It might be that your external service can't call this repeatedly. To which I say:

• Don't rely on the external service, have your own server call it regardless, even if there's no work to do
• A standard WP Cron task would also work
• Run it every 5 minutes

• Make the task call itself if there's still stuff to do, using a nonblocking request. This way it'll keep spawning new instances until it's finished e.g.

          if ( $count === 5 ) {
              wp_remote_get( home_url('?mbimport=true&key=abc'), [ 'blocking' => false ]);
              exit;
          )
  

GUI?

If you want a progress meter for a UI in the dashboard, just count how many jpeg files are left in the folder to import. If you need to configure it, then build a UI and save the settings in options, then pull from options in the CLI script.

Have You Considered Using the REST API?

Sidestep the entire process and add the files via the REST API. You can do a POST request to example.com/wp-json/wp/v2/media to upload the jpegs. No code on your site necessary

https://stackoverflow.com/questions/37432114/wp-rest-api-upload-image

Answer 2

There is already a built-in function created just for that purpose. You don't need to write walls of codes to uploads images from your disk. You can use media_sideload_image instead.

This function will upload your files, take care of the file name, date, ID and the rest of the stuff.

I haven't tested this with absolute path (It needs the URL), but it would be easy to convert absolute path to URLs, based on your skill in writing the above script.

// Set the directory
$dir = ABSPATH .'/wpse';
// Define the file type
$images = glob($directory . "*.jpg");
// Run a loop and upload every file to media library
foreach($images as $image) {
    // Upload a single image
    media_sideload_image($image,'SOME POST ID HERE');
}

That's all you need. Images must be attached to a post, but you can detach them afterward.

Answer 3

Your server may also be limiting the total upload capacity, it not just what you set in the code. check with your provider or if you have access to the WHM, change the maxium php upload for the account (usually that is 8GB) and you will have to change the php.ini . The following URL was helpful to me when I had to do the same thing:

http://www.wpbeginner.com/wp-tutorials/how-to-increase-the-maximum-file-upload-size-in-wordpress/

However, it is not a complete solution if the server you are on has a limit.