I manage a small site that has a very active comment area. The site is on shared hosting. There is a daily post entry, then a robust comment area. It is not untypical to have more than 100 comments on a day's post.

The result is that loading or refreshing the current day's page is very slow - up to or more than 30 seconds. Sometimes a page load will time out with a 500 error. Other times a comment 'submit' will result in a 500 error, and the comment is lost.

The site gets under 5K visits a day, with less than 30 concurrent users during the day. Some of those visits are individuals re-loading the page to see the latest comments.

Lately, the search engines have been indexing the site. I see this in the GA real-time monitor as page loads from past years. Each search index page load also displays that day's comments, which put an additional load on the comment query for that page.

The daily comments area is a popular feature of the site, so do not want to disable that. The increased costs in changing from a shared server to a dedicated server are not possible (there is no revenue generated for the site). Blocking specific search engines via the robots.txt or htaccess files won't work - the search engine locations are variable and change. The theme uses the standard loop for displaying the comments.

I tried a caching plugin (WP Cache), but it didn't seem to reduce the page load time. (Not to mention that it shortened the 'remember me' in the comment entry form to 1 day, which irritated the commenters and me.)

So what is the best way to increase the 'efficiency' of the site's page load time with the active discussion environment on the site?

I manage a small site that has a very active comment area. The site is on shared hosting. There is a daily post entry, then a robust comment area. It is not untypical to have more than 100 comments on a day's post.

The result is that loading or refreshing the current day's page is very slow - up to or more than 30 seconds. Sometimes a page load will time out with a 500 error. Other times a comment 'submit' will result in a 500 error, and the comment is lost.

The site gets under 5K visits a day, with less than 30 concurrent users during the day. Some of those visits are individuals re-loading the page to see the latest comments.

Lately, the search engines have been indexing the site. I see this in the GA real-time monitor as page loads from past years. Each search index page load also displays that day's comments, which put an additional load on the comment query for that page.

The daily comments area is a popular feature of the site, so do not want to disable that. The increased costs in changing from a shared server to a dedicated server are not possible (there is no revenue generated for the site). Blocking specific search engines via the robots.txt or htaccess files won't work - the search engine locations are variable and change. The theme uses the standard loop for displaying the comments.

I tried a caching plugin (WP Cache), but it didn't seem to reduce the page load time. (Not to mention that it shortened the 'remember me' in the comment entry form to 1 day, which irritated the commenters and me.)

So what is the best way to increase the 'efficiency' of the site's page load time with the active discussion environment on the site?

Added

I have used the Query Monitor plugin to look at query times. It still seems that the queries related to the comments table are the main issue in slowing down page response. (Page response takes about 8-10 seconds to show a page with comments.)

Splitting the comments into pages irritated the regular visitors on the site. And it didn't seem to improve page load time.

Would partitioning the comments table help? Only the comments from the last 5 days need optimization. I don't worry about search indexing time to get old posts. Just the response time for looking at the latest post.

Or is there an additional index that might be added to the comments table to make it's queries more efficient?