How to periodically scrape and cache strings from remote txt files. - My First Plugin

Question

I maintain a resource website, FilterLists, which is essentially a directory of txt files hosted around the net. I am trying to build my first WordPress plugin to (a) learn and (b) add a new feature to this tool.

I am trying to scrape the last modified time of each of the linked txt files (as provided in the actual text by the authors). As it stands, my first pass at the plugin works properly, but does all of the scraping and parsing at the time a visitor (or, in my case, the CDN every few hours) requests the site, which takes way to long to do for all of the files listed. Ideally, I'd like to periodically ping each of the files at regular intervals probably using something like wp-cron (which I am not familiar with outside of knowing it exists) and caching the results. If I can get all of the Last Modified dates updated once a day or so, that would be sufficient.

Here's the plugin so far, and you can see it working live on the first 5 txt links on the FilterLists site linked above.

Can anyone point me in the right direction to [have wp-cron] scrape the dates periodically rather than at page request time?

I'm pretty new to PHP and WP development, so verbosity is welcomed.

Thanks.

Answer 1

The Transient API may help you here. A transient is variable stored for a defined amount of time.

function filemtime_remote( $url )
{
    $list = file_get_contents( $url , null , null , 0 , 200);
    $important = explode("Last modified: ",$list)[1];
    $mydate = substr($important, 0, 21);
    return $mydate;
}

The above is your current code which seeks the "Last modified" time each time a page is loaded.

You can convert it to:

function filemtime_remote( $url ){   
        # Get the transient
        $mydate = get_transient( 'lm_' . esc_url( $url ) );

        if( false === $mydate ) {
            # The transient expired or does not exist, so we fetch the last modified again
            $list = file_get_contents( $url , null , null , 0 , 200);
            $important = explode("Last modified: ",$list)[1];
            $mydate = substr($important, 0, 21);

            # We then save that date in a transient(which we can prefix with something like "lm_")
            # We are saving the transient for 12 hours
            set_transient( 'lm_' . esc_url( $url ), $mydate , 12 * HOUR_IN_SECONDS );
        }

        return $mydate;
    }

I haven't tried it, but the logic is here: Use get_transient() to see if we have a value recorded with the last 12 hours. If we do have a value (the returned value will not be FALSE), use that value. If we do not have a value(the returned value will be FALSE), then request the data and then save it in a transient, using set_transient(), that will expire in 12 hours.