I have just received an email from Google advising that it can't accessing certain javascript and css content from my site.
I've looked at the robots.txt file which contains:
User-agent: * Crawl-delay: 5
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-*
It looks like it is the Disallow: /wp-*
that is doing the damage.
I am just going through the process of disabling each plugin in turn to see which one (if any) is causing this line to appear in the robots.txt file, but could there be another reason for it (e.g., core WordPress feature/setting)?
And is it fine and safe for me to just remove this Disallow: /wp-*
line?