2

I am trying to upload many hundreds of images daily from a folder on the server to the media library using the following script that is scheduled via CRON:

<?php
require_once('../../../../public/wordpress/wp-load.php');
require_once('../../../../public/wordpress/wp-admin/includes/image.php');

function importImage($imagePath, $postId)
{
    $succeededFileCount = 0;
    $failedFileCount = 0;
    $files = scandir($imagePath);

    foreach ($files as $file) {
        if (in_array($file, ['.', '..'])) {
            continue;
        }

        $newPath = $imagePath . "/" . $file;
        $filePath = realpath($newPath);

        if (is_dir($newPath) && $item != '.' && $item != '..' && $item != 'failed_files') {
            importImage($newPath, $postId);
        } elseif ($item != '.' && $item != '..' && $item != 'failed_files') {
            $filename = basename($file);
            $uploadFile = wp_upload_bits($filename, null, file_get_contents($filePath));
            $wp_upload_dir = wp_upload_dir();

            if (! $uploadFile['error']) {
                $fileType = wp_check_filetype($filename, null);
                $attachment = [
                    'guid' => $wp_upload_dir['url'] . '/' . basename( $filename ),
                    'post_mime_type' => $fileType['type'],
                    'post_parent' => $postId,
                    'post_title' => preg_replace('/\.[^.]+$/', '', $filename),
                    'post_content' => '',
                    'post_status' => 'inherit'
                ];
                $attachmentId = wp_insert_attachment($attachment, $uploadFile['file'], $postId);

                if (! is_wp_error($attachmentId)) {
                    $attachmentData = wp_generate_attachment_metadata($attachmentId, $uploadFile['file']);
                    wp_update_attachment_metadata($attachmentId, $attachmentData);
                }
            } else {
                echo '<span style="color: red; font-weight: bold;">Error: ' . $uploadFile['error'] . '</span>';
            }


            if ($attachmentId > 0) {
                $succeededFileCount++;
                echo '<span style="color: green; font-weight: normal;">File import succeeded: ' . $filePath . "</span><br />";

                if (! unlink($filePath)) {
                    echo '<span style="color: red; font-weight: bold;">Unable to delete file ' . $filePath . " after import.</span><br />";
                }

                $page = get_post($postId);

                if ($page->post_content) {
                    $content = $page->post_content;
                    $start = strpos($content, "[gallery ") + strlen("[gallery ");
                    $end = strpos(substr($content, $start), "]");
                    $shortcode = substr($content, $start, $end);
                    $attrs = shortcode_parse_atts($shortcode);
                    $attrs["ids"] .= "," . $attachmentId;
                    $tempIds = explode(",", $attrs["ids"]);
                    $tempIds = array_filter($tempIds);
                    rsort($tempIds);
                    $attrs["ids"] = implode(",", $tempIds);
                    $shortcode = "";

                    foreach ($attrs as $key => $value) {
                        if (strlen($shortcode) > 0) {
                            $shortcode .= " ";
                        }

                        $shortcode .= $key . "=\"" . $value . "\"";
                    }

                    $newContent = substr($content, 0, $start);
                    $newContent .= $shortcode;
                    $newContent .= substr($content, $start + $end, strlen($content));
                    $page->post_content = $newContent;
                    wp_update_post($page);
                }
            }
        }
    }

    echo $succeededFileCount . " files uploaded and imported successfully. <br />";
    echo $failedFileCount . " files failed to uploaded or import successfully.";
}

get_header();

if (get_option('rmm_image_importer_key') != urldecode($_GET['key'])) {
    echo '<div id="message" class="error">';
    echo "<p><strong>Incorrect authentication key: you are not allowed to import images into this site.</strong></p></div>";
} else {
    echo '<br /><br />';
    $mtime = microtime();
    $mtime = explode(" ", $mtime);
    $mtime = $mtime[1] + $mtime[0];
    $starttime = $mtime;
    $dataset = get_option('rmm_image_importer_settings');
    if (is_array($dataset)) {
        foreach ($dataset as $data) {
            if (isset($data['folder'])
                || isset($data['page'])) {
?>
    <h2>Import from folder: <?php echo $data['folder']; ?></h2>
    <p>
<?php
                importImage(realpath(str_replace('//', '/', ABSPATH . '../../' . $data['folder'])), $data['page']); ?>
    </p>
<?php
            }
        }
    }
    $mtime = microtime();
    $mtime = explode(" ", $mtime);
    $mtime = $mtime[1] + $mtime[0];
    $endtime = $mtime;
    $totaltime = ($endtime - $starttime);
    echo 'Files imported to media library over ' . $totaltime . ' seconds.<br /><br />';
}

get_footer();

The problem is that no matter what I do, the script fails after just two images with this error:

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 28672 bytes) in /home/forge/morselandcompany.com/public/wordpress/wp-includes/wp-db.php on line 1841

I have the memory limit set to 1024M in wordpress, as well as PHP. I don't see why this script should need more than 128M, really. How can this script be optimized to work properly? (Average image size is 800kB.)

Some initial debugging with BlackFire.io suggests the following memory hogs: - wpdb->query: 3.2MB, called 31 times - mysqli_fetch_object: 1.89MB, called 595 times - run_init() in wp-settings.php: 5.4MB, called once In total blackfire suggests that over 8MB is required to run this script!

I have also tested with all plugins disabled, and that ended with the same result.

I am running on - PHP 7.1
- Ubuntu 16.04
- DigitalOcean VPS (1 CPU, 1GB RAM)
- Wordpress 4.8
- NGINX 1.11.5

Thanks for any help!

Update: in the interest of completeness so that others may utilize the solution for memory leaks associated with get_post and wp_update_post, I have posted the finalized code that solved the problem above. As you can see, the solution was to write my own queries using $wpdb instead of relying on the two WP methods causing the memory leaks:

<?php

require_once('../../../../public/wordpress/wp-load.php');
require_once(ABSPATH . 'wp-admin/includes/media.php');
require_once(ABSPATH . 'wp-admin/includes/file.php');
require_once(ABSPATH . 'wp-admin/includes/image.php');

function importImage($imagePath, $postId)
{
    $succeededFileCount = 0;
    $failedFileCount = 0;
    $files = scandir($imagePath);

    foreach ($files as $file) {
        if (in_array($file, ['.', '..'])) {
            continue;
        }

        $newPath = $imagePath . "/" . $file;
        $filePath = realpath($newPath);

        if (is_dir($newPath) && $file != 'failed_files') {
            importImage($newPath, $postId);
        } elseif ($file != 'failed_files') {
            $webPath = str_replace($_SERVER['DOCUMENT_ROOT'], '', $imagePath);
            $imageUrl = str_replace('/wordpress', '', get_site_url(null, "{$webPath}/" . urlencode($file)));
            $imageUrl = str_replace(':8000', '', $imageUrl);
            $attachmentId = media_sideload_image($imageUrl, 0, '', 'id');

            if ($attachmentId > 0) {
                $succeededFileCount++;
                echo '<span style="color: green; font-weight: normal;">File import succeeded: ' . $filePath . "</span><br />";

                if (! unlink($filePath)) {
                    echo '<span style="color: red; font-weight: bold;">Unable to delete file ' . $filePath . " after import.</span><br />";
                }

                global $wpdb;
                $page = $wpdb->get_results("SELECT * FROM wp_posts WHERE ID = {$postId}")[0];

                if (is_array($page)) {
                    $page = $page[0];
                }

                if ($page->post_content) {
                    $content = $page->post_content;
                    $start = strpos($content, "[gallery ") + strlen("[gallery ");
                    $end = strpos(substr($content, $start), "]");
                    $shortcode = substr($content, $start, $end);
                    $attrs = shortcode_parse_atts($shortcode);
                    $attrs["ids"] .= "," . $attachmentId;
                    $tempIds = explode(",", $attrs["ids"]);
                    $tempIds = array_filter($tempIds);
                    rsort($tempIds);
                    $attrs["ids"] = implode(",", $tempIds);
                    $shortcode = "";

                    foreach ($attrs as $key => $value) {
                        if (strlen($shortcode) > 0) {
                            $shortcode .= " ";
                        }

                        $shortcode .= $key . "=\"" . $value . "\"";
                    }

                    $newContent = substr($content, 0, $start);
                    $newContent .= $shortcode;
                    $newContent .= substr($content, $start + $end, strlen($content));
                    $wpdb->update(
                        'post_content',
                        ['post_content' => $newContent],
                        ['ID' => $postId]
                    );
                }
            }
        }
    }

    echo $succeededFileCount . " files uploaded and imported successfully. <br />";
    echo $failedFileCount . " files failed to uploaded or import successfully.";
}

get_header();

if (get_option('rmm_image_importer_key') != urldecode($_GET['key'])) {
    echo '<div id="message" class="error">';
    echo "<p><strong>Incorrect authentication key: you are not allowed to import images into this site.</strong></p></div>";
} else {
    echo '<br /><br />';
    $mtime = microtime();
    $mtime = explode(" ", $mtime);
    $mtime = $mtime[1] + $mtime[0];
    $starttime = $mtime;
    $dataset = get_option('rmm_image_importer_settings');

    if (is_array($dataset)) {
        foreach ($dataset as $data) {
            if (isset($data['folder'])
                || isset($data['page'])) {
?>
    <h2>Import from folder: <?php echo $data['folder']; ?></h2>
    <p>
<?php
                importImage(realpath(str_replace('//', '/', ABSPATH . '../../' . $data['folder'])), $data['page']); ?>
    </p>
<?php
            }
        }
    }
    $mtime = microtime();
    $mtime = explode(" ", $mtime);
    $mtime = $mtime[1] + $mtime[0];
    $endtime = $mtime;
    $totaltime = ($endtime - $starttime);
    echo 'Files imported to media library over ' . $totaltime . ' seconds.<br /><br />';
}

get_footer();
  • Why isn't this a WP CLI command called directly by Cron instead of a standalone file bootstrapping WP that anybody can trigger – Tom J Nowell Jul 15 '17 at 0:26
  • Because it is also triggered externally by applications living outside the server (they hit this url after FTPing images to the server, to monitor the import progress and get immediate notification if anything fails). In the future I plan to implement your solution below. – mike.bronner Jul 15 '17 at 3:20
3

A few things

  • Use media_handle_sideload so that WordPress moves the files to the right location and validates them for you, creates the attachment post etc, none of this manual stuff
  • Don't run this once and expect it to do everything. You're just going to run into the same problem but further into the import. If you give it infinite memory you'll have a time limit execution problem where the script simply runs out of time
  • Process 5 files at a time, and repeatedly call run it until nothing is left to process
  • Use a WP CLI command, don't bootstrap WordPress and call it from the GUI. Call it from Cron directly and skip the pinging a URL business. CLI commands get unlimited time to do their job, and you can't call them from the browser. The GET variable with the key becomes completely unnecessary.
  • Escape escape escape, you're echoing out these arrays and values, assuming what they contain is safe, but what if I snuck a script tag in there? Unable to delete file <script>...</script> after import.. Biggest security step you can take that makes the most difference yet the least used

A Prototype WP CLI command

Here is a simple WP CLI command that should do the trick. I've not tested it but all the important parts are there, I trust you're not a complete novice when it comes to PHP and can tighten any loose screws or minor mistakes, no additional API knowledge necessary.

You'll want to only include it when in a WP CLI context, for example:

if ( defined( 'WP_CLI' ) && WP_CLI ) {
    require_once dirname( __FILE__ ) . '/inc/class-plugin-cli-command.php';
}

Modify accordingly, dumping the command in functions.php of a theme and expecting it all to work will cause errors as WP CLI classes are only loaded on the command line, never when handling a browser request.

Usage:

wp mbimport run

Class:

<?php
/**
 * Implements image importer command.
 */
class MBronner_Import_Images extends WP_CLI_Command {

    /**
     * Runs the import script and imports several images
     *
     * ## EXAMPLES
     *
     *     wp mbimport run
     *
     * @when after_wp_load
     */
    function run( $args, $assoc_args ) {
        if ( !function_exists('media_handle_upload') ) {
            require_once(ABSPATH . "wp-admin" . '/includes/image.php');
            require_once(ABSPATH . "wp-admin" . '/includes/file.php');
            require_once(ABSPATH . "wp-admin" . '/includes/media.php');
        }

        // Set the directory
        $dir = ABSPATH .'/wpse';
        // Define the file type
        $images = glob( $dir . "*.jpg" );
        if ( empty( $images ) {
            WP_CLI::success( 'no images to import' );
            exit;
        }
        // Run a loop and transfer every file to media library
        // $count = 0;
        foreach ( $images as $image ) {
            $file_array = array();
            $file_array['name'] = $image;
            $file_array['tmp_name'] = $image;

            $id = media_handle_sideload( $file_array, 0 );
            if ( is_wp_error( $id ) ) {
                WP_CLI::error( "failed to sideload ".$image );
                exit;
            }

            // only do 5 at a time, dont worry we can run this
            // several times till they're all done
            $count++;
            if ( $count === 5 ) {
                break; 
            }
        }
        WP_CLI::success( "import ran" );
    }
}

WP_CLI::add_command( 'mbimport', 'MBronner_Import_Images' );

Call repeatedly from a real cron job. If you can't, then either use WP Cron, or have a hook on admin_init that checks for a GET variable. Use the code inside the run command with some modifications.

When WP CLI Isn't An Option

Using a standalone PHP file that bootstraps WP is a security risk and a great target for attackers if they want to exhaust your server resources ( or trigger duplication issues by hitting the URL multiple times all at once ).

For example:

// example.com/?mbimport=true
add_action( 'init', function() {
    if ( $_GET['action'] !== 'mbimport' ) {
        return;
    }
    if ( $_GET['key'] !== get_option('key thing' ) ) {
        return;
    }
    // the code from the run function in the CLI command, but with the WP_CLI::success bits swapped out
    // ...
    exit;
}

Repeated Calling

It might be that your external service can't call this repeatedly. To which I say:

  • Don't rely on the external service, have your own server call it regardless, even if there's no work to do
  • A standard WP Cron task would also work
  • Run it every 5 minutes
  • Make the task call itself if there's still stuff to do, using a nonblocking request. This way it'll keep spawning new instances until it's finished e.g.

            if ( $count === 5 ) {
                wp_remote_get( home_url('?mbimport=true&key=abc'), [ 'blocking' => false ]);
                exit;
            )
    

GUI?

If you want a progress meter for a UI in the dashboard, just count how many jpeg files are left in the folder to import. If you need to configure it, then build a UI and save the settings in options, then pull from options in the CLI script.

Have You Considered Using the REST API?

Sidestep the entire process and add the files via the REST API. You can do a POST request to example.com/wp-json/wp/v2/media to upload the jpegs. No code on your site necessary

https://stackoverflow.com/questions/37432114/wp-rest-api-upload-image

  • Thank you, Tom for your excellent description and explanation of a possible solution! While certain constraints limit me from 8mplementing it all the moment, I will keep it in mind the next time maintenance is performed for this client. I particularly like the progress bar idea, that will work very well. – mike.bronner Jul 15 '17 at 3:17
  • @mike.bronner Thanks for the bounty, however I would have suggested you to grant it to tom's answer if I have seen it earlier,since it's more detailed. – Jack Johansson Jul 15 '17 at 3:33
  • @mike.bronner copy the code out of the run method, I added an example – Tom J Nowell Jul 15 '17 at 10:32
  • Thanks, Tom! You're too kind! I would donate you 50 pts as well, if I could. marked yours as the accepted answer, as it covers everything. – mike.bronner Jul 15 '17 at 16:16
3
+50

There is already a built-in function created just for that purpose. You don't need to write walls of codes to uploads images from your disk. You can use media_sideload_image instead.

This function will upload your files, take care of the file name, date, ID and the rest of the stuff.

I haven't tested this with absolute path (It needs the URL), but it would be easy to convert absolute path to URLs, based on your skill in writing the above script.

// Set the directory
$dir = ABSPATH .'/wpse';
// Define the file type
$images = glob($directory . "*.jpg");
// Run a loop and upload every file to media library
foreach($images as $image) {
    // Upload a single image
    media_sideload_image($image,'SOME POST ID HERE');
}

That's all you need. Images must be attached to a post, but you can detach them afterward.

  • Thank you for providing a concise possible solution. Is there a way to sideload images without adding them to a post? The reason my code above is so long, is that I am adding images to galleries in posts, but not as attachments to posts. – mike.bronner Jul 13 '17 at 17:55
  • @mike.bronner It's temporarily. You can run another foreach right after the firs one is finished, and detaching the attachments from posts. – Jack Johansson Jul 13 '17 at 17:56
  • 1
    Thanks, Jack! Let me give this a shot and I will report back with my findings! :) Wish me success! – mike.bronner Jul 13 '17 at 17:57
  • Hi Jack, reporting back. While your solution was more elegant than mine, and I was able to get it working, the memory culprits seem to be the get_post and wp_update_post functions. I replaced those with $wpdb queries and everything is working wonderfully now on 128M memory limit. However, thanks for responding and prompting the troubleshooting that allowed me to find the solution. (I will award the bounty to you once the time limiter allows me to.) – mike.bronner Jul 13 '17 at 20:07
  • 1
    Yea, the sideload function was definitely the way to go. :) I had written this years ago when the sideload function wasn't as versatile. The memory leaks also only started appearing recently (maybe since WP 4.5). Before that everything was working great. – mike.bronner Jul 14 '17 at 0:04
-1

Your server may also be limiting the total upload capacity, it not just what you set in the code. check with your provider or if you have access to the WHM, change the maxium php upload for the account (usually that is 8GB) and you will have to change the php.ini . The following URL was helpful to me when I had to do the same thing:

http://www.wpbeginner.com/wp-tutorials/how-to-increase-the-maximum-file-upload-size-in-wordpress/

However, it is not a complete solution if the server you are on has a limit.

  • Thanks Debbie. In this instance that was not the case, I control the server completely, as I am running a VPS. – mike.bronner Jul 15 '17 at 3:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.