3

I am developing a WordPress plugin and wants to run a background task that calls an API and updates database table. Now, the API can only give results for 5 DB entries in one go and for 500 entries in my table, I have to make 100 API call. The API has allowed TPS quota of 1 and also in every 40 minutes, its old response expires which means I need to update my table if any entry is older than 40 minutes by making a new API call. And, all these DB entries have to be shown to the page viewer with latest data.

The solution that I came up with is scheduling a cron task that runs every minute and does API calls one after another for 25 seconds and then dies so that it doesn't exceed PHP max execution time limit. And at time a customer comes, he has not to wait for API call or be throttled by API

But the problem is I can't rely on wp-cron as it will be called only when page hit occurs(not like actual cron and since I am plugin developer I can't schedule a system cron on my customer's WordPress hosting environment). Also, if the person receives only one page hit in 3 hours, that first person in 3 hours has to either wait for those API calls to finish(not at all desirable or better to say feasible) or else he will not be shown the data.

Is there any other way to solve this problem of updating the DB entries via some background process that is neither slowing down the client and nor depending on page load. So even if 4 clients come in a day they all get latest data (data updated within last 40 minutes) as its already been updated in the background?

Improve this question
4
  • if you don't have enough visitors, you can set a linux cron on another server which execute curl -F http://the-site-URL every hour.
    – mmm
    Feb 25, 2018 at 19:00
  • As already mentioned, I am a plugin owner and not website owner. I ship my plugin to different WordPress users with different hosting environment config and for most environments, I will not be allowed to set up Linux cron.
    – beginner
    Feb 25, 2018 at 19:05
  • Then you cannot make more than explain in your plugin to the website owner that he must set an external cron if his website doesn't have enough visitors.
    – mmm
    Feb 25, 2018 at 19:08
  • you can also recommand to your users to use a webcron
    – mmm
    Feb 25, 2018 at 19:12

2 Answers 2

Reset to default
1

There is no reason for your design, but the core of your problem is that (I assume) people are trying to use a free tier of the API instead of properly paying for its use and removing the ridiculous restrictions.

Anyway, the solution is fairly simple, run your cron every half an hour, allocate to your import process unlimited time (and you might need to instruct your users to increase memory limit), and just loop until you got all the requested data with a second delay http://php.net/manual/en/function.sleep.php between each request.

Even the worst case of each request taking 5 seconds to complete, the whole thing should be done in 5 minutes.

As for the user view, you should keep new info in some kind of limbo state until you got all of it, and only then make it presentable.

Improve this answer
7
  • By allocating unlimited time, you mean(set_time_limit(0))? But it wouldn't be allowed on all hosting environments. Adding to top of that, my plugin users are not very technically sound people and they will not be able to give permissions to set_time_limit or increase memory limit?
    – beginner
    Feb 25, 2018 at 19:32
  • yes was referring to set_time_limit(0). If it is not allowed how do those sites do any importing at all? Anyway, you can not satisfy all the crazy hosting with all their crazy short comings as the core of your problem is probably the attempt to not pay for the use of the API which will probably let you get all the data in one simple request
    – Mark Kaplun
    Feb 25, 2018 at 19:58
  • The core of my problem is to serve the latest_content(age<=40 minutes)to the website visitors. Now, since API has allowed TPS of 1(which it's happy to provide), I need too much time to update my DB by doing API calls. Also, that too will happen when someone comes on the website, so for a website with average 2 hit per hour, the first user will not see proper content and the only second user will see cached content due to wp-cron's nature of not behaving as actual cron.
    – beginner
    Feb 26, 2018 at 2:38
  • In short, I want to know if there is any way by which: 1. I can call the background process in every 40 minutes even if the website has 5 page hits in a day? 2. Implement long-running background process by either breaking task to smaller subtasks(5 API calls in every task to avoid max execution time limit)? Can, libraries like wp-background-processing help here?
    – beginner
    Feb 26, 2018 at 2:42
  • no, your core problem, in addition to the API design is that you are trying to satisfy people that have no control on their server, and various limitations on using them. Hosting companies that limit excessive use of server resources will most likely not be happy with handling a request per second, no matter how it is being triggered, especially since you are going to do DB writes, and will call it excessive. There is no way to win in this kind of scenario.
    – Mark Kaplun
    Feb 26, 2018 at 3:18
0

Disable WP Cron from running on users' traffic. Then run cron through a cron or timed job somewhere else. You can achieve this by adding this to your wp-config.php

define('DISABLE_WP_CRON', true);

Then set up a cron job to hit this URL - https://domain.com/wp-cron.php?doing_wp_cron at your desired time intervals. A quick wget code below:

wget -q -O - https://domain.com/wp-cron.php?doing_wp_cron >/dev/null 2>&1

I suggest every minute, as all other cron jobs in the WordPress installation will be dependent on it

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.