Transferring/Uploading Data from DB to WordPress

Question

I've started to work on a project regarding WordPress Website, I have some experience with WordPress, but it's not on the level I want it to be in the incoming months.

I'm more of a Python Developer then Wordpress Developer, so my question may be a bit noobie.

The website was organized around scraping and providing those scraped data to end-users. So I started to rebuild the scraped because it had too many bugs, and here my question begins, the last script was written in PHP, and my scraper is being built in Python.

So I am wondering, when script output the data, the user will need to upload it to WordPress, is there any better more automated solutions, where I can store it in some DB, and then just migrate data from that DB to the WordPress DB, but I've looked few things online, but it's mostly some add-ons that do not give any answers.

Has someone did something like this, any advice would be helpful.

Answer 1

Since you are building the scraping functionality outside of WordPress, you will have to use a way that allows you to add contents to WordPress from the outside as well. I basically see three options:

1. Update via the REST API - Using the REST API you can post updates to the WordPress contents. You can do this right from your Python scraper code, posting to the REST API directly, when there is a change in the data. For example, you can create a post or update a post. The REST API is capable of handling
2. Read the external database from WordPress - Alternatively, you could write a script that periodically checks the database where the Python scraper stores its data, for new changes. You could for example write a WP-CLI script that runs every x minutes (using a cron). That WP-CLI script in turn reads the external database and uses native WordPress functions like wp_insert_post() and wp_update_post() to save the data to WordPress.
3. Build functionality to live read external database - Depending on what kind of data you are working with, importing the data might not be suitable. You could also make a page template that reads the external database while it is requested. The data doesn't have to be stored in WordPress in order to be shown in WordPress pages. In this case, you would consider the database to be an external source, like an API, from where you fetch data as you need it.

Implementations like this often get really specific, so these are just pointers of the routes you could go. Depending on the type of data, the volume of the data and the interval in which the data is updated, one of these routes should suit your use case best.