0

I get a lot of 404 errors on my WordPress site and I have controlled a lot of those using the Redirection plugin.

Now there is a particular type of 404s that originate from Googlebot requests. These are for my older posts and try to look for files related to my old theme and plugins which I don't use anymore. If I browse those pages in the browser via link there is no 404 thrown, so it is like Google bot is still looking at old pages.

These plugin & theme files vary a lot. Some are js, while others are CSS files. I read somewhere earlier that overtime Google bot will learn from these 404 errors and old posts in Google cache will get updated. But after many months I still see these type of 404s happening.

I have submitted sitemaps multiple times to Google webmaster tool as well. I don't like seeing these 404 errors and I get like 500 of these every month.

How can I solve this problem? Is there any way to tell Google bot that these plugins/themes are not used by me so stop looking for them and crawl the newer version of pages if you will.

Here is one such post from my 404 log https://www.bintooshoots.com/nokia-3310-4g-connectivity-android-based-yunos-spotted-tenaa/

Improve this question

1 Answer 1

Reset to default
2

I know - I hate the fact that Google indexes the site code, and then floods my logs with 404s for old files.

You can add something like this to your robots.txt file:

User-agent: *
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/

You could probably just have Disallow: /wp-content/ but that would also stop your images in the uploads folder from being indexed too... unless you want that!

Improve this answer
2
  • Thanks for the great tip. My yoast seo plugin was managing the robots.txt I thought but they seem to have made changes according to this recent post. yoast.com/wordpress-robots-txt-example Still I know this blocking is needed to reduce the 404 mess so I would go with this for now.
    – Mr.Hunt
    Commented Jul 31, 2018 at 9:33
  • Just an update that adding Disallow: /wp-content/themes/ makes Google complain about site not being mobile friendly and fails their check mobile-friendly site test. So I removed it.
    – Mr.Hunt
    Commented Aug 9, 2018 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.