Today I received an email from Webmaster Tools stating that "Googlebot cannot access CSS and JS files". So I set about updating my robots.txt file located in the root of my server.
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/themes/ #<--CAUSE OF THE PROBLEM Sitemap: http://example.com/sitemapindex.xml
I removed the line "Disallow: /wp-content/themes/" from the file and re-uploaded the file but when I viewed the new file in my browser, nothing had changed I then did the following:
1) Deleted the .txt file from the root
2) Cleared cache (W3 Total Cache)
But still robots.txt was viewable in my browser. This could only mean one thing, that the file was being virtually generated right? I even viewed in a different browser to ensure no caching was taking place.
After much searching I have found no answer as to why my robots.txt won't update or where it is being generated from?
Does anyone here have any ideas as to whats going on?
Some plugins I use:
1) Better WordPress Google XML Sitemaps 2) W3 Total Cache 3) Yoast SEO
Any help would be much appreciated, thanks.