6

I have a feed aggregator and I want will have 2 million posts. I am forced to host the content in order for the posts to show in the loop.

A friend stated I need to use a third-party search engine and upgrade to 8G RAM.

Improve this question
1
  • the only answer to this kind of question is "yes". with this amount of details there can not be any better answer.
    – Mark Kaplun
    Jul 21, 2017 at 6:36

4 Answers 4

Reset to default
11

2,000,000,000,000 WordPress posts, is it possible?

Simple answer, Yes. The days of 2GB size limit on file systems are gone. Now, you can literally have unlimited entries on almost everywhere if your hardware can support it. So I'm gonna change the question into this:

2,000,000 WordPress posts, Should I have it?

To decide whether WordPress is a good choice for such projects, you have to note a few things here:

1- Uploads

If your posts are going to have images, then this is where you might have a problem. considering the default thumbnail sizes of every image (thumbnail, medium, medium-large, large + the image itself), you will end up having at least 5 generated image for each post (assuming each post only has 1 image). This means you will end up having 10,000,000 images, and if you not choose to organize your uploads, well then, trouble gets closer.

Although you can have 4,294,967,295 (232 -1) files in a folder even on a NTFS partition (the actual limitation is lower), but having drastic decreased performance while listing 10 million files is inevitable. So I suggest you should use a upload organizer plugin in this case.

2- Database entries

Databases can also contain massive number of rows and huge size of entries. A MyISAM storage engine supports 232 rows per table, which is far far more than what average websites need.

WordPress creates entries other than the original post itself in the database. Each post can have unlimited meta attached to it, limited by the database itself. So if your posts are going to have a lot of meta data (Images, Custom fields, etc) then doing a heavy task such as a complicated meta query is gonna take considerable time and resources.

It is worth mentioning that you should clean up your database once in a while, to avoid unwanted performance decrease. You can either optimize it using scripts such as PHPMyAdmin or by using WordPress plugins.

Conclusion

WordPress itself is not the most optimized platform. If you do a quick search, you'll notice there are a lot of questions about decreased performance when it comes to thousands and tens of thousand of posts.

So, again it's up to you to decide if this platform is suitable for you or not. There are a lot of scripts already made for this. A quick search about RSS Aggregator can lead you to endless result on this. You might as well find it best to have your own script written, for a couple of thousand dollars, if you are going commercial.

Improve this answer
5

Adding a little Rick Hellewell answer.

Creating some custom PHP/MySQL code to do it is a different thing than using Wordpress for it - the main difference is the table structure, which can be much more efficient when designed for a specific purpose. Wordpress is design to keep all data in mainly one or two tables so getting them out usually needs a lot of SQL JOINs which could be avoided in custom designed site. More efficient means less resources (RAM, processor) needed.

Of course it depends how complicated will be your aggregated post data - if just a simple data that you can store in a basic Wordpress post that could be OK, but if you start to add custom fields to it - the things can get complicated with this kind of amounts.

If you need more than a simple Wordpress post and you'd like to stay with Wordpress after all, and you don't need Wordpress'es back-end for accessing/editing the aggregated posts I would recommend creating some custom table for storing and getting them (especially if you plan to have such a lot of them and then using) the wpdb class for working with that table(s).

Improve this answer
3

You sure can. Wordpress doesn't limit the quantity of posts, taxonomies, users etc.. you can have on your site. It all depends on the capability of your host service.

Improve this answer
1

Years ago, I created a feed aggregator in PHP/MySql. It would CURL sites on a regular basis, comparing their latest post with the latest stored in a table. The latest posts (excerpts) were displayed on demand (main page load). The site is now gone (for various reasons), but the concept worked.

You could do the same thing with WP. You would just need to get the data from the external source, then create a new post with some back-end code. Then WP could be used to display the posts (or a subset based on some criteria like a category that you set with your data grab).

But, as Cesar says, you are going to need a robust hosting plan (dedicated servers, maybe cloud-based content) to do it correctly.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.