Refactoring Wordpress to improve memory performance [closed]

Question

I had a close look at Wordpress memory consumption. On my site, it seems that for each page hit 20MB of RAM gets allocated, just to prepare comfy environment for all the plugins to run in. I plotted it thusly:

There is no single spot to optimize, no single bad guy who eats most of the memory. Consumption is all spread over many many many php modules.

How can we make Wordpress initialize its environment in memory only once, and then reuse it many times for each hit? I do not want to have slow PHP eat 20 MB at each user click - even on a server with lots of memory, it takes seconds to get all that work done. You would basically need read-only chunks of memory that can be reused.

Also... why 20MB? Can anyone provide insights into this?

Edit: Here is WinCacheGrind output on the Wordpress running on my development machine (vastly faster than the shared hosting). As you can see, it takes over a second of crunching just to produce the main page's HTML. Slow that down by shared hosting and you have a recipe for trouble. I picked the method that takes most of the time. How would you go about optimizing this?

Edit: Here are query statistics from this fantastic functions.php profiling tool.

Load: 12 queries - 532ms - 19.1MB - 43 cache hits / 53
Query: 15 queries - 563ms - 19.0MB - 72 cache hits / 86
Display: 21 queries - 705ms - 19.2MB - 234 cache hits / 257

Edit: Do you want to see something guaranteed to freak you out? Insert these lines at the end of index.php:

echo "<pre>\n";
print_r(get_defined_vars());
echo "</pre>\n";

I tried to count how many times is the current post's body stored in the memory. I counted 20 instances. Then I realized that PHP has reference counting, so the amount of copies reduced to just three: two seem to be in WP_Query, one in the object cache. I am investigating further.

This is why I think WordPress is in need of refactoring targeting the memory issues. You can no longer blame its memory consumption on sheer complexity of what it does. It simply does a bunch of things wrong.

Edit: After a day of trying to figure this out, here are my findings:

1) 88% of all memory comes from require or include or include_once type of calls:

2) The php file includes happen mostly during the first part of serving a request (not surprisingly), which is also where all the memory is eaten up:

3) It is quite interesting to plot all the functions that are being executed during making a request. There are over 12000 calls total. I jittered them to make it more visible (the Level axis is basically depth of stack):

4) The only way forward that I can think of is minimizing the amount of .php files included. If I split the functions per file they came from, you can see that many files are hit once or twice at most. We need a way how to skip those when they are not needed. For instance my remote database backup plugin gets loaded and registered, just to be never used at all. Here is the above plot split by file name:

I am offering a bounty worth all my reputation :) for refactorings that would lead to cutting my blogs memory footprint by 30% or more.

Edit: I installed WP 3.1, here is comparison with the old version.

Blue is WP 3.1, red is 3.0.4. The new WP is faster, but also eats more memory.

Here is a list by include file.

This let me realize just how much memory is eaten by "All In One SEO pack" - one avenue would be to use only a fraction of the plugin's functionality to get what I want. Also, my own plugins seem to be pretty bad.

I would like to try conditional loading on e.g. comment.php (I disallow comments on my blog) and several others. I deleted all the deprecated code. I trimmed down kses.php to only load its global tables on demand. I simplified l10n (I do no localization), making its functions return the strings right away, without lookups. I am still far from the 30% mark I arbitrarily set up.

Edit: I downloaded and enabled APC with default settings (32MB of opcode cache). Here is the comparison:

You can see that the code loading accelerated massively, and the code also takes less space in memory (probably because we deal just with opcodes, not the original source). The memory consumption is however still quite high.

Answer 1

Not worth the trouble. WordPress doesn't eat a lot of memory just-because. It eats a lot of memory because it runs a lot of functionality under the hood.

It is far more easier and efficient to cache results (page generated) with static cache plugin and serve that. That way most visitor will not even hit WP itself.

Answer 2

And this is why I think WordPress is in serious need of a rewrite. You can no longer blame its memory consumption on sheer complexity of what it does. It simply does things wrong.

What a naive conclusion. Read Things You Should Never Do, Part I.

Thanks for the memory usage plots, though.

Much later edit: Autommatic has released a library called prefork that seems to do what you're asking for: loading the WordPress code in RAM only once.

Answer 3

Starting with WordPress 3.2, PHP 5.2 will be the minimum requirement. I think with that under our belts, bits of the core can start to be restructured, and use classes with auto-loading. This would let us avoid loading some chunks of code unless they were actually needed. For example, if there were no embeds or galleries in a page view, we might be able to avoid loading a lot of the media code.

However, even if they decide to go that route, I would expect it to be a slow evolution (like much of the other under-the-hood changes that have occurred). It would require shuffling around the location of a lot of files and code, which could possibly break backwards compat for some plugins.

Part of the problem (if it can really be called that) is that without that kind of conditional loading, the core framework can't know ahead of time which functionality it will need or not need in order to generate the content view. So a lot of functions have to be loaded up just in case they're needed.

Answer 4

How can we make Wordpress initialize its environment in memory only once, and then reuse it many times for each hit?

It's called opcode-caching.

http://en.wikipedia.org/wiki/PHP_accelerator

Answer 5

you probably won't manage to reduce ram usage that much. But if you're using mod_php, you may want to switch to mod_fcgid instead.

while mod_php is slightly slower, it loads php even when it doesn't need to, such as serving images, static files, or even caching. If you have lot of requests, this is lot of ram.

using fcgid will reduce this a lot.

also, using a static cache (like w3total cache) will avoid calling php at all which is a really great advantage: less ram usage, less db connections.

Answer 6

Ha. I'm working on a web app now that I fully intend to overload with data and usage beyond what my shared hosting account can handle, so I decided - while it would have been super easy to build in WP - to try working from BackPress as a framework and build out only what I needed for my specific use-cases.

So I've been able to reduce my core environment from the hundreds of PHP files in WP to only the twenty or so that I actually need, while still being able to leverage all the db, HTTP, user-management, formatting, and cron functions that I love in WordPress.

The problem is that its a lot of work, and I would never trust my hackjob for anything beyond my own personal use. If you want to use the full WP environment, take it as it is. Its as good as it is because of hundreds of developers fine-tuning it over several years. Like everyone here has said, you'll get a lot farther by finding a better hosting plan and researching caching techniques than you're likely to get by hacking the core.

Answer 7

Yes, WordPress loads up everything first and then does what we ask it to do. I can recall somewhere that we can create a virtual pool in RAM where we can put in files. I had the idea of putting the whole WordPress in memory (<10MB) & then we can save a lot of I/O which alone should give a speed boost. But I never got the chance to try it and moreover I am not really that much proficient in pursuing something like that. But it looks worth a try.

Answer 8

a few basic suggestions :

1. w3 total cache plugin for caching..
2. get memcache installed and enabled, also enable from w3 total cache settings (opcode cache is a good option too but it doesn't go well with w3 total cache plugin)
3. minimize queries to direct links in theme files..
4. Disable all extra unused plugins and remove.
5. optimize Database.

i am running a well known wordpress site with huge traffic daily.. im not on dedicated even, doing great for me :)