How to avoid timeout when importing a lot of data?

Question

I need to import catalog data from external server. The data is arranged in categories and then items.

the API requires the following calls: 1. Get categories 2. Get category items 3. get item data

this task works once a day by using a cron job. the catalog contains about 5000 items.

items import is encapsulated into 3 files: 1. get cats - gets all categories 2. get category items - gets all the items, creates each item (cpt) and than calls 3 3. get item info - updates the item data (custom fields)

no matter what I do, I'm getting at some point error 504 Gateway Time-out.

How can I bypass that ?

script 1:

    <?php 
/* Short and sweet */
define('WP_USE_THEMES', false);
require('./wp-blog-header.php');
set_time_limit(600);

$processed_cats = array();
$processed_items = array();

$cats = get_terms( 'svod_cat', array('hide_empty' => false));
foreach ($cats as $cat) {
    if (get_field('disabled', 'svod_cat_'.$cat->term_id))
        continue;

    $cat_id = get_field('cat_id', 'svod_cat_'.$cat->term_id);
    $curl = curl_init();
    curl_setopt ($curl, CURLOPT_URL, 'http://example.com/import_cat.php?cat_id='.$cat->term_id);
    curl_exec ($curl);
    curl_close ($curl);
    echo $res."<br/>";
}
svod_update_disabled_items($processed_items);


?>

script 2:

    <?php 
/* Short and sweet */
define('WP_USE_THEMES', false);
require('./wp-blog-header.php');
set_time_limit(3600);
error_reporting(E_ERROR | E_WARNING | E_PARSE);

$wp_cat = $_GET['cat_id'];

function svod_get_items_list($cat) {
    $url = 'http://clientserver.com/getmyitems.php?cat=$cat';
    $curl = curl_init();
    curl_setopt ($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
    $items_json = curl_exec ($curl);
    curl_close ($curl);
    $items = json_decode($items_json);
    return $items->results;
}

function svod_get_items($svod_cat, $wp_cat) {
    global $processed_items;
    $new = 0;
    $update = 0;
    $items = svod_get_items_list($svod_cat);
    echo "Got items list with ".count($items)." items<br/>";
    flush();
    foreach ($items as $item) {
        if (($my_item = get_page_by_title( $item->VodTitle, OBJECT, 'svod_item')) == NULL) {
            $id = svod_create_new_item($item, $wp_cat);
            $new++;
        } else {
            $id= svod_update_item($my_item, $item, $wp_cat);
            $update++;
        }
        $url = "http://example.com/import_item.php?item_id=$id";
        $curl = curl_init();
        curl_setopt ($curl, CURLOPT_URL, $url);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
        $res = curl_exec ($curl);
        curl_close ($curl);
        $processed_items[] = $id;
        echo $item->vodId . " ".$res."<br/>";
        flush();
    }
    echo "Done import; created $new, updated $update<br/>";
    flush();
}

function svod_create_new_item($item, $wp_cat) {
    $my_post = array(
          'post_title'    => $item->VodTitle,
          'post_content'  => '',
          'post_status'   => 'publish',
          'post_author'   => 1,
          'post_type'     => 'svod_item'
        );

    $pid = wp_insert_post($my_post);
    if (!is_wp_error($pid)) {
        wp_set_post_terms( $pid, array($wp_cat), 'svod_cat', true);
        update_field('field_55ec1ed404c0e', $item->VodId, $pid);
    }
    return $pid;
}

function svod_update_item($my_item, $item, $wp_cat) {
    wp_set_post_terms( $my_item->ID, array($wp_cat), 'svod_cat', true);
    update_field('field_55ec1ed404c0e', $item->VodId, $pid);
    return $my_item->ID;
}

$processed_items = array();

echo "Importing category ".$wp_cat.'<br/>';
$cat_id = get_field('cat_id', 'svod_cat_'.$wp_cat);
svod_get_items($cat_id, $wp_cat);

?>

script 3:

    <?php 
/* Short and sweet */
define('WP_USE_THEMES', false);
require('./wp-blog-header.php');
set_time_limit(60);
error_reporting(E_ERROR | E_WARNING | E_PARSE);

function svod_get_item_info($vodid) {
    $url = 'https://clientserver.com/get_info.php?id='.$id;

    $curl = curl_init();
    curl_setopt ($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
    $item_json = curl_exec ($curl);
    curl_close ($curl);
    $item = json_decode($item_json);
    return $item;
}

function svod_update_item($pid, $info) {

    $my_post = array(
      'ID'           => $pid,
      'post_content' => $info->VodDesc,
    );

    // Update the post into the database
    wp_update_post( $my_post );

    update_field('field_55ec08a2bc9bf', $info->VodReleaseYear, $pid );
    update_field('field_55ec08a9bc9c0', $info->VodRating, $pid);
    update_field('field_55ec08c4bc9c1', $info->VodRunTime, $pid);
    update_field('field_55ec091bbc9c2', $info->VodPosterUrl, $pid);
    update_field('field_55ec15f4bc9c3', ($info->VodHD == 'true'), $pid);
}

$wp_vod_id = $_GET['item_id'];
$vod_id = get_field('vod_id', $wp_vod_id);
$info = svod_get_item_info($vod_id);
svod_update_item($wp_vod_id, $info);
echo "<br/>Item updated !";
?>

Answer 1

For avoid timeout best option is wp-cli. Wp-cli is wordpress command line interface. you can export/import data using wp-cli.

Answer 2

I would split those operations into several actions. If it's impossible to finish it in one run, there won't be any better solution. The main thing is if the error is caused by client server timeout (unable to download all data in time), or your server timeout (unable to process all data in time). Another uknown thing is what operation is the most time consuming. Script 3?

I would create some custom DB tables. Then run cronjob to download all categories and save them to one table, then another cronjob to download items list. And then cronjob, that takes every 5 minutes for example 10 items from list, and download its "item info"... until every item has its info. In that moment you have all "source" data from client server.

After this, you can run another operation to process saved data and import them to your WP (probably not all in bulk, but again splitted into more runs). Processed items will be removed from table, and when it's empty, import is DONE. If some run fails/timeouts, items will stay in DB table, and waits until another run. Nothing will be lost.

Answer 3

set_time_limit(0) will disable the time limit check and should solve the problem if that is the actual issue. You can also try and sent multiple requests at the same time with curl which should make things faster.