I maintain a resource website, FilterLists, which is essentially a directory of txt files hosted around the net. I am trying to build my first WordPress plugin to (a) learn and (b) add a new feature to this tool.

I am trying to scrape the last modified time of each of the linked txt files (as provided in the actual text by the authors). As it stands, my first pass at the plugin works properly, but does all of the scraping and parsing at the time a visitor (or, in my case, the CDN every few hours) requests the site, which takes way to long to do for all of the files listed. Ideally, I'd like to periodically ping each of the files at regular intervals probably using something like wp-cron (which I am not familiar with outside of knowing it exists) and caching the results. If I can get all of the Last Modified dates updated once a day or so, that would be sufficient.

Here's the plugin so far, and you can see it working live on the first 5 txt links on the FilterLists site linked above.

Can anyone point me in the right direction to [have wp-cron] scrape the dates periodically rather than at page request time?

I'm pretty new to PHP and WP development, so verbosity is welcomed.


up vote 1 down vote accepted

The Transient API may help you here. A transient is variable stored for a defined amount of time.

function filemtime_remote( $url )
    $list = file_get_contents( $url , null , null , 0 , 200);
    $important = explode("Last modified: ",$list)[1];
    $mydate = substr($important, 0, 21);
    return $mydate;

The above is your current code which seeks the "Last modified" time each time a page is loaded.

You can convert it to:

function filemtime_remote( $url ){   
        # Get the transient
        $mydate = get_transient( 'lm_' . esc_url( $url ) );

        if( false === $mydate ) {
            # The transient expired or does not exist, so we fetch the last modified again
            $list = file_get_contents( $url , null , null , 0 , 200);
            $important = explode("Last modified: ",$list)[1];
            $mydate = substr($important, 0, 21);

            # We then save that date in a transient(which we can prefix with something like "lm_")
            # We are saving the transient for 12 hours
            set_transient( 'lm_' . esc_url( $url ), $mydate , 12 * HOUR_IN_SECONDS );

        return $mydate;

I haven't tried it, but the logic is here: Use get_transient() to see if we have a value recorded with the last 12 hours. If we do have a value (the returned value will not be FALSE), use that value. If we do not have a value(the returned value will be FALSE), then request the data and then save it in a transient, using set_transient(), that will expire in 12 hours.

  • thanks for that! that makes sense. I will try it later on tonight. Since this function is called via a shortcode, could I maybe use a random (between reasonable limits, like 6-18 hours) transient timeout for each url so that, in theory, all of the urls will not have to be hit at the same time for one unlucky visitor every 12 hours? Also, will the $mydate variable work properly for all instances of the shortcode calling this function? Or would I somehow need an array of variables to store the unique transient value for each instance of the shortcode? – Collin M. Barrett Feb 29 '16 at 20:07
  • It worked and is now live on the first half of the lists on FilterLists! Still testing, but this got me going. Thanks! – Collin M. Barrett Feb 29 '16 at 21:14

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.