I manage a small site that has a very active comment area. The site is on shared hosting. There is a daily post entry, then a robust comment area. It is not untypical to have more than 100 comments on a day's post.

The result is that loading or refreshing the current day's page is very slow - up to or more than 30 seconds. Sometimes a page load will time out with a 500 error. Other times a comment 'submit' will result in a 500 error, and the comment is lost.

The site gets under 5K visits a day, with less than 30 concurrent users during the day. Some of those visits are individuals re-loading the page to see the latest comments.

Lately, the search engines have been indexing the site. I see this in the GA real-time monitor as page loads from past years. Each search index page load also displays that day's comments, which put an additional load on the comment query for that page.

The daily comments area is a popular feature of the site, so do not want to disable that. The increased costs in changing from a shared server to a dedicated server are not possible (there is no revenue generated for the site). Blocking specific search engines via the robots.txt or htaccess files won't work - the search engine locations are variable and change. The theme uses the standard loop for displaying the comments.

I tried a caching plugin (WP Cache), but it didn't seem to reduce the page load time. (Not to mention that it shortened the 'remember me' in the comment entry form to 1 day, which irritated the commenters and me.)

So what is the best way to increase the 'efficiency' of the site's page load time with the active discussion environment on the site?