How to break down importing of feeds

Question

I am attempting to update a large array of feeds into WordPress, using fetch_feeds. As can be expected, when going over 50 feeds the server is starting to time out. I would still like to update the feeds in bulk but without crashing the server.

Can you suggest a method through which I can break things down? Maybe using AJAX? Is that the best technique? Any examples I can look at?

Answer 1

You can try using wp cron functions, programing a feed import every 10 minutes or so, until you have nothing left in queue.

Example code:

<?php
register_activation_hook( __FILE__, 'wp1903_schedule_cron' );

function wp1903_schedule_cron() {
    $counter = 0;
    $feeds = array(
        'http://example.com/feed1',
        'http://example.com/feed2',
        'http://example.com/feed3',
        'http://example.com/feed4'
    );

    foreach ($feeds as $feed) {
        $counter++;
        // schedule every 10 mins
        $time = time() + (600 * $counter);
        wp_schedule_single_event( $time, 'wp1903_process_feed', $feed );
    }
}

add_action('wp1903_process_feed', 'wp1903_import_feed');

function wp1903_import_feed($feed_url = '') {
    // do the import
}

It is essential only to run the scheduler on a activation hook, or else at every page load the crons will be rescheduled.

Answer 2

You've tried to extend the script's maximum execution time with set_time_limit()? This will not cause the server to crash (because it has a memory limit too):

set_time_limit( 0 ); // No time limit is imposed

I'm not sure what you mean with "breaking down" the script, you might want to have a second thought on that.

What you could do if there are to many rows in the feed's XML you also could make your own import script and take care of its handle your own way by processing small part by small part.

$xml_handler = new ProductsParser();
$xml_parser  = xml_parser_create();

$source = '--URL--';

xml_set_object( $xml_parser, $xml_handler );
xml_parser_set_option( $xml_parser, XML_OPTION_CASE_FOLDING, false );
xml_set_element_handler( $xml_parser, 'startElement', 'endElement' );
xml_set_character_data_handler( $xml_parser, 'cdata' );

$fp = fopen ( $source, 'r' );
while ( $chunk = fread( $fp, 4096 ) ) {
  xml_parse( $xml_parser, $chunk, feof( $fp ) );
  flush();
}

fclose( $fp );

class ProductsParser {
  public $product; # Holds the record values
  public $elem_key; # Holds the current element key while reading
  public $record; # Holds the record tag

  function __construct( $args ) {
    $this->product  = false;
    $this->elem_key = false;
    $this->record   = '--RECORD-NAME--';
  }

  function startElement( $parser, $tag, $attributes ) {
    if ( $this->record == $tag && ! is_array( $this->product ) ) {
      $this->product = array();
    } elseif( is_array( $this->product ) ) {
      $this->elem_key = $tag;
    }
  }

  function endElement( $parser, $tag ) {
    if ( $this->record == $tag && is_array( $this->product ) ) {
      // Process the product row

      $this->product = false;
    } elseif ( is_array( $this->product ) && $this->elem_key != false ) {
      $this->elem_key = false;
    }
  }

  function cdata( $parser, $cdata ) {
    if ( is_array( $this->product ) && $this->elem_key != false ) {
      $this->product[$this->elem_key] = $cdata;
    }
  }
}

More information about the XML parser can be found in the PHP manual.

Answer 3

When it comes down to high execution time scripts I usually split it up as simple as possible using ajax so here is a concept class that should get you started with creating an admin page and ajaxed import loop:

<?php
class Ajax_Import{

    public function __construct(){
        if(is_admin()){
            add_action('admin_menu', array($this, 'add_plugin_page'));
        }
        add_action('wp_ajax_do_feed_import',array($this,'do_import'));
    }

    public function add_plugin_page(){
        // This page will be under "Settings"
        add_options_page('ajaxed Import', 'ajaxed Import', 'manage_options', 'test-setting-admin', array($this, 'create_admin_page'));
    }

    public function create_admin_page(){
        wp_enqueue_script( 'jquery');
        ?>
        <div class="wrap">
            <?php screen_icon(); ?>
            <h2><?php _e('AJAXed Import'); ?></h2>          
            <table class="form-table">
                <tbody>
                    <th scope="row">
                        <label for="my-text-field"><?php _e('Feeds To Import'); ?></label>
                    </th>
                    <td>
                        <textarea id="feeds"></textarea>
                        <br>
                        <span class="description"><?php _e('Enter Each feed in a new line.'); ?></span>
                    </td>
                </tbody>
            </table>
            <p class="submit"><input type="submit" id="do_import" value="Save Changes" class="button-primary" name="Submit"></p>
            <div id="import_status" style="display:none"></div>
        </div>
        <?php
        $this->import_loop_js();
    }

    public function do_import(){
        //do your import here ex:
        $rss = fetch_feed(esc_url( $_POST['feed_url']));
        if (!is_wp_error( $rss ) ) {
            $count = $rss->get_item_quantity(); 
            //do actuall import loop or whatever
            echo json_encode(array('message' => '<div id="message" class="success"><p>imported ' . $count . ' items from '.esc_url( $_POST['feed_url']).'</p></div>'));
        }else{
            echo json_encode(array('errors' => '<div id="message" class="error"><p>' . $rss->get_error_message() . '</p></div>'));
        }
        die();
    }

    public function import_loop_js(){
        ?>
        <script type="text/javascript">
        jQuery(document).ready(function($){
            var ajax_import = (function(){
                var instantiated;
                var feeds;
                function init (){
                    return {
                        start_import: function(){

                            instantiated.feeds = $("#feeds").val().split('\n');
                            instantiated.update_status('', true);
                            instantiated.next_feed();
                        },
                        make_ajax_call:function(feed){
                            data = {
                                action: "do_feed_import",
                                feed_url: feed
                            };
                            $.post( 
                                ajaxurl,
                                data, 
                                function(res){
                                    if (res.error){//add error message
                                        instantiated.update_status(res.error);
                                    }else{//add success message
                                        instantiated.update_status(res.message);
                                    }
                                    //call next feed
                                    instantiated.next_feed();
                                }, 
                                'json'
                            );
                        },
                        next_feed:function(){
                            if (instantiated.feeds.length > 0) {//if we have feeds in the list starts ajax calls
                                instantiated.make_ajax_call(instantiated.feeds.shift());
                            } else{
                                instantiated.update_status("all Done!!!");
                                $("#do_import").removeAttr("disabled");
                            }
                        },
                        update_status:function(m,clear){
                            clear = clear || false;
                            if (clear)
                                $("#import_status").html('').show();
                            else
                                $("#import_status").append(m);
                        }
                    }
                }

                return {
                    getInstance :function(){
                        if (!instantiated){
                            instantiated = init();
                        }
                        return instantiated; 
                    }
                }
            })();
            $("#do_import").on('click',function(){
                $(this).attr('disabled','disabled');
                ajax_import.getInstance().start_import();
            });
        });
        </script>
        <?php

    }
}

$Ajax_Import = new Ajax_Import();