0

In one of my site, i am using the SERVER cron to get the data from the third party API vai CURL every minute and run the set transient function. It is working fine most of the fine. But when there are many traffic to the site, the DATABASE gets down.

The main problem is the size of the data coming from third party API. It returns the ARRAY data of nearly 1300 products.

Can somebody share the better idea to properly managing the every minute CRON for large data coming from API?

  • Are 1300 products changing that often that you need to refresh them every 60 seconds? – Jacob Peattie Mar 7 '18 at 5:22
  • Yes. They are gold silver related pricing. We need to update the pricing of each products as matching as to the current market rates of gold/silver. – prasidhda Mar 7 '18 at 5:39
  • You probably shouldn't be using a transient. The wp_options table, including transients are all loaded on every page load, so you're loading all the data on every page load regardless of whether or not its needed. You should store the data in a custom table and only query the bits you need when you need it. – Jacob Peattie Mar 7 '18 at 6:37
  • There will be no "autoload" for transient type of options. So, whether we use the custom table or transient, i don't think there will be much difference. – prasidhda Mar 7 '18 at 6:49
  • Sorry, you're right about autoload. But it you want the data you still need to load the whole transient. Are you creating 1300 separate transients or storing the whole response in one transient? – Jacob Peattie Mar 7 '18 at 6:50
1

There is no way to do that. You need to design a push solution instead of a pull one, and if you can't and must have to use pull.... well than it sucks to be you :(.

Web servers and web software is not designed to be used for "heavy lifting" in such frequent interval. You need to rethink your requirements. For example, just to process 1300 items can easily take more than a minute, especially on a busy server.

One path of possible solution, is to have an update server which is not your wordpress app server (you can run wordpress on it if it makes coding easier, but the only thing that server should do is to update the DB, probably should use WP-CLI for that, and not a web server).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.