Import all media referenced in posts

Question

I want to import into the media library, all the images & videos referenced in my blog posts.

1. Can I use get_attached_media() to get the images referenced on a page, even if they're not formally attached?
2. Should I use media_handle_sideload() to insert the media into the library?

Background

I've moved hosts, all my blog posts are available and all my images are in /wp-content/uploads/2001/08 etc.

I have around 13,000 images across ~120 subfolders (10 years of blogging ☺).

Here's what I've tried that hasn't worked:

• Import WXR - imported the posts, but wouldn't import the images which were on the same server.
• Media From FTP - didn't work with the huge number of images.
• Add from Server - only works on a per-file basis. I couldn't select a whole folder and all its sub-folders.
• Other plugins seem to suffer from the same issue - they don't deal with nested directories, and they can't cope with thousands of files.

All I want is something to scan my posts, find any <img src="https://example.com/..., then import that into the media library - leaving the media in the same location. Attachment to the post & regeneration of thumbnails would be a bonus, but not essential.

Answer 1

1. Can I use get_attached_media() to get the images referenced on a page, even if they're not formally attached?

No, if they're not attached they won't be attached. I assume you're talking about images in your posts that are offsite?

If so, you need to download them, create an attachment post, attach it to the post, then replace the URL in the conten

2. Should I use media_handle_sideload() to insert the media into the library?

Yes, the sideloading functions will even create the attachment post and attach it to a given parent ID, although finding the images to sideload, and replacing the URLs is another matter.

Answer 2

I ended up using WP-CLI and a bit of bash to fix this.

1. Get all post IDs into a file
   • ./wp-cli.phar post list --field=ID > posts.txt
2. Generate HTML from each post and put it in separate file
   • cat posts.txt | while read line ; do ./wp-cli.phar post get $line --field=post_content > test/$line.html ; done
3. Use PHP's DOMdocument to extract all the images and generate the WP-CLI commands

<?php

$start = "https://example.com/blog/";

foreach (glob("*.html") as $file) {
    if($file == '.' || $file == '..') continue;
    $id = substr($file, 0, -5);
    $html = file_get_contents($file);
    $doc = new DOMDocument();
    $doc->loadHTML($html);
    $images = $doc->getElementsByTagName('img');

    foreach($images as $img) {
       $imgSrc = $img->getAttribute('src');
       $starter = substr( $imgSrc, 0, strlen($start) );
       if($starter == $start) {
          $imgLocation = substr( $imgSrc, strlen($start) );
          echo "./wp-cli.phar media import {$imgLocation} --post_id={$id} --skip-copy\n";
       }
    }
}

• Run as commands.php > commands.txt

Run all the WP-CLI commands in the file

• bash commands.txt

A little convoluted, but it worked!