Creating 20,000 Posts or Pages using a .csv file?

Question

Here's my challenge: I have a .csv file which contains 20,000 rows of data.

Each row contains 5 columns of data on an particular individual.

I need each row (each person) to be populated into a WordPress Page or Post. So, in other words, I need to convert 20,000 rows from a .csv into 20,000 posts or pages (i.e. HTML files).

How would I go about this? Is there a plugin or would you suggest that this is more of a mySQL command?

Any direction gratefully received!

Thanks!

Answer 1

Import posts from a .csv file with WP-CLI

If our import.csv is tab delimited, with two columns:

Planet Mars     The red planet. 
Planet Earth    Our blue planet.

then we can parse it with help of this answer, where we use IFS=$'\t' instead as tab delimiter:

while IFS=$'\t', read col1 col2; \
do \
    echo "$col1|$col2";
done < import.csv

Note that the column count must match the file's columns.

Now let's use WP-CLI to create posts with wp create post:

while IFS=$'\t', read col1 col2; \
do \
  wp post create \
    --post_title="$col1" \
    --post_content="$col2" \
    --post_author=1 \
    --post_type=post \
    --post_status=publish \
done < import.csv

If we have additional meta data, like average distance from the Sun in astronomical units (au):

Planet Mars     The red planet.     1.52
Planet Earth    Our blue planet.    1.00

then we can try:

while IFS=$'\t', read col1 col2 col3; \
do \
  wp post create \
    --post_title="$col1" \
    --post_content="$col2" \
    --post_author=1 \
    --post_type=post \
    --post_status=publish \
    | xargs -I % wp post meta add % distance_au "$col3"; done < import.csv

It would be handy if wp post create supported the meta_input argument of wp_insert_post().

We could also have written this out with wp_insert_post() from a php file with the wp eval-file command.

WP-CLI supports importing users from a csv file, but I haven't seen this kind of core feature for posts yet.

We could instead write our own WP-CLI command for csv posts import.

Then there's are plugins, like WP CLI Import CSV, that seems to have done it already (I'm not related to it and haven't used it).

Note that WP-CLI can import wxr files, if you can convert your csv file into wxr file(s).

Also note that currently (4.8) WordPress doesn't handle hierarchical post types (aka pages) well in the wp-admin backend. All posts, of that type, are loaded at once into the parent selection's dropdown. That doesn't scale well and can result in PHP timeout, when working in the backend! See for example here. So go for non-hierarchical post types (aka posts) in your large imports.