Web Scraping with Cron

Question

I'm currently scraping prices from three websites using xPath, but since it updates on every page load, it makes loading slow. What I would like to do is store that data and only update it weekly. So:

1) Scrape prices from three different websites using xPath,

2) Store that data (and update it weekly with Cron)

At the moment, I am successfully scraping the prices with xPath using the following:

<?php // a new dom object
$store_link = get_field('store_link');

$ch = curl_init($store_link);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$cl = curl_exec($ch);

$dom = new DOMDocument();
@$dom->loadHTML($cl);
$xpath = new DOMXpath($dom);

$price = $dom->getElementById("tr_base_main_price");
$store_price = trim(str_replace(str_split('$'), '', ($price->nodeValue)));
} ?>

I'm completely in the dark with making it work with cron though. I'm using ACF, so I know I would use update_field(), but I'm not sure where to go from there. My current reasoning is:

1) Have xPath and update_field() function in page.php (function XYZ)

2) Schedule function XYZ in functions.php

Is that the correct way to do it? Any guidance will be GREATLY appreciated!!

Thanks so much!

Answer 1

You don't necessarily need to use the WP Cron API for this. Instead you could modify your existing code to make use of WP Transients. Transients are basically Options that expire after a set time. You can wrap your current scrape process in an if statement that checks for the existence of the stored transient. If it exists, it pulls the data from the transient. If it doesn't, it performs the scrape and stores the data as a transient.

$store_price = get_transient( "store_price_{$post->ID}" );
if ( ! $store_price ) {
  // curl & xpath stuff here
  set_transient( "store_price_{$post->ID}", $store_price, WEEK_IN_SECONDS );
}
// do stuff with $store_price here

Transients API: http://codex.wordpress.org/Transients_API

Answer 2

You can use wp_schedule_event() to fire your price update function at set intervals!