robots.txt Not Updating

Question

Today I received an email from Webmaster Tools stating that "Googlebot cannot access CSS and JS files". So I set about updating my robots.txt file located in the root of my server.

Original Contents:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/ #<--CAUSE OF THE PROBLEM

Sitemap: http://example.com/sitemapindex.xml

I removed the line "Disallow: /wp-content/themes/" from the file and re-uploaded the file but when I viewed the new file in my browser, nothing had changed I then did the following:

1) Deleted the .txt file from the root

2) Cleared cache (W3 Total Cache)

But still robots.txt was viewable in my browser. This could only mean one thing, that the file was being virtually generated right? I even viewed in a different browser to ensure no caching was taking place.

After much searching I have found no answer as to why my robots.txt won't update or where it is being generated from?

Does anyone here have any ideas as to whats going on?

Some plugins I use:

1) Better WordPress Google XML Sitemaps
2) W3 Total Cache
3) Yoast SEO

Any help would be much appreciated, thanks.

Answer 1

Normally, if there's a WordPress file on disk, that'll be served first directly by Apache or Nginx, before WordPress gets involved.

This is done in your virtualhost config, e.g. in Nginx you'll typically find the following, which tells it to try actual files first before letting index.php handle the URL and generate a page on demand.

location / {
    index index.php index.html;
    try_files $uri $uri/ /index.php?$args;
}

So if your robots.txt is being ignored you may have something wrong with your webserver configuration.

I just tried those three plugins you mentioned with the twentyfifteen theme and everything was working ok. Yoast SEO lets you edit robots.txt from the Admin pages by the way (go to SEO > Tools > File Editor.)

If there's no robots.txt file found, or control has been passed to WordPress, the default output is:

User-agent: *
Disallow: /wp-admin/

See wp-includes/functions.php to see how this works, but don't ever edit core files.

This can be customised with actions and filters – for example the BWP Sitemaps plugin adds a Sitemap: line.

If you can't find a plugin (or anything in your theme) that's hooking into do_robots or do_robotstxt, this is a hack for your theme's functions.php file that will probably do the job:

/**
 * Remove unwanted 'themes' from robots.txt with find/replace
 * @param string $input original robots.txt content
 *
 * @return string mixed
 */
function patch_robots_file( $input ) {
    return str_replace( 'Disallow: /wp-content/themes/', '', $input );
}

add_filter( 'robots_txt', 'patch_robots_file' );