Disallow: /wp-* in robots.txt?

Question

I have just received an email from Google advising that it can't accessing certain javascript and css content from my site.

I've looked at the robots.txt file which contains:

User-agent: * Crawl-delay: 5

Disallow: /feed/

Disallow: /trackback/

Disallow: /wp-admin/

Disallow: /wp-includes/

Disallow: /xmlrpc.php

Disallow: /wp-*

It looks like it is the Disallow: /wp-* that is doing the damage.

I am just going through the process of disabling each plugin in turn to see which one (if any) is causing this line to appear in the robots.txt file, but could there be another reason for it (e.g., core WordPress feature/setting)?

And is it fine and safe for me to just remove this Disallow: /wp-* line?

Answer 1

It seems to be a WP default setting, as many Webmasters have gotten this warning and never edited the robots.txt. Removing all the disallows is the easiest solution, but I assume you want some or all of those directories blocked.

Google is only concerned about the .js and .css files, so you could in theory edit the robots.txt to include:

User-Agent: Googlebot Allow: /.js Allow: /.css

However, being that specific could require future changes to the user agent, in case more search crawlers follow Google's example.

You want to make sure you know how robots.txt work so you don't accidentally block your entire site or important sections. Here is a good reference for more details about robots.txt:

http://www.robotstxt.org/robotstxt.html

Answer 2

I have sorted this now. I'm not sure where the original robots.txt content could have come from(??), but I have now changed it on the origin server to:

User-agent: * Crawl-delay: 5

Disallow: /feed/

Disallow: /trackback/

Disallow: /wp-admin/

Disallow: /wp-includes/

Disallow: /xmlrpc.php

Disallow: /wp-content/

Disallow: /wp-*

Also, the site uses a CDN, so I specified a seperate set of rules for the CDN's robots.txt file:

User-agent: *

Allow: /wp-content/

Disallow: /