I maintain a resource website, FilterLists, which is essentially a directory of txt files hosted around the net. I am trying to build my first WordPress plugin to (a) learn and (b) add a new feature to this tool.
I am trying to scrape the last modified time of each of the linked txt files (as provided in the actual text by the authors). As it stands, my first pass at the plugin works properly, but does all of the scraping and parsing at the time a visitor (or, in my case, the CDN every few hours) requests the site, which takes way to long to do for all of the files listed. Ideally, I'd like to periodically ping each of the files at regular intervals probably using something like wp-cron (which I am not familiar with outside of knowing it exists) and caching the results. If I can get all of the Last Modified dates updated once a day or so, that would be sufficient.
Here's the plugin so far, and you can see it working live on the first 5 txt links on the FilterLists site linked above.
Can anyone point me in the right direction to [have wp-cron] scrape the dates periodically rather than at page request time?
I'm pretty new to PHP and WP development, so verbosity is welcomed.