Add robots.txt to root

Question

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.

I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but http://mysite.com/robots.txt still returns a 404.

Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?

Answer 1

First of all, in order for Wordpress to generate a robots.txt for you you must be using a non-default permalink structure. Make sure you've selected an option in the Settings > Permalinks menu.

Also, if a robots.txt file exists at your root directory it will override the setting in Wordpress. It looks like you already have a robots.txt file and that is the reason the wordpress setting is ignored.

Answer 2

You don't need to add robots.txt file to the root of your site. robots.txt file is generated in real time, when you visit http://mysite.com/robots.txt. The function, responsible for creation of this file, is do_robots.

If you wish to add your own directives, just write your hook for robots_txt filter, like this:

add_filter( 'robots_txt', 'wpse8170_my_robots_txt', 10, 2 );
function wpse8170_my_robots_txt( $output, $public ) {
    if ( '0' != $public ) {
        $output .= '
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
';
    }
    return $output;
}