I can only relate my own experience, and so far, I have not found a "definite" rule to fix all issues at one stroke.
The major problem with DreamHost's setup is that, in the eternal fight to keep memory consumption at a minimum, it means getting rid of as many features as possible — namely, all that will reduce bandwidth (good for the visitors!) or CPU (good for the server, but DreamHost doesn't control CPU consumption as aggressively as they control RAM). For instance, this means getting rid of gzip'ed HTML + CSS (which will consume CPU + RAM) or any of the several Minify plugins (which will consume RAM as well). The more sophisticated the cache (I'm fond of using W3 Total Cache, or at least WP Super Cache), the more RAM will be consumed as well.
Similarly, many plugins which limit the number of MySQL queries to improve performance will instead consume RAM. So finding a trade-off where you can still keep your site replying with good performance while avoiding to consume precious RAM is a hard task!
So far, my best results on busy sites is to uncheck Page Speed Optimization and Extra Web Security which will apparently consume a lot of RAM, and rely instead on a combination with W3 Total Cache and Cloudflare (free reverse proxy service). Cloudflare will effectively do the same thing as the "Extra Web Security" module, but since it runs outside DreamHost, it's fine. W3 Total Cache consumes a lot of memory but once the pages are statically stored locally, Cloudflare will very efficiently cache them — so you might get 404/500 while editing posts, at least your visitors will not experience them (Cloudflare also can serve static pages even if DreamHost gives a 404 or a 500).
Also, thanks to this article, I have found out that FastCGI uses more RAM than 'normal' CGI. And since PHP 5.3 is better at managing RAM (more aggressive garbage collection, less memory leaks), I've experimentally switched to PHP 5.3 CGI (not FastCGI) without Page Speed Optimization nor Extra Web Security, relying on W3 Total Cache + Cloudflare to accelerate the site. Now the backoffice is slower (more CPU consumption!) but at least I don't see 404/500 (so far!).
I'm still unhappy with the combination, so I'll certainly continue to tweak DreamHost's settings hoping to reduce RAM consuption even more and still get adequate performance. Like @dgw said, I also use a lot of plugins — because I require their functionality. Not everybody hosting WP with DreamHost has simple, blogging needs; the more complex the site, the more functionality it will require... and that's the beauty of WordPress, you just need to use the plugins you really need, and keep the core WP install simple if you're happy with few needs. Plugins, however, aren't necessarily "bad" or that heavy on the site; but it's true that some may consume a lot of RAM...