Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.

I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but http://mysite.com/robots.txt still returns a 404.

Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?

share|improve this question
    
I just realized that the "Discourage search engines from indexing this site" setting adds a <meta content="noindex,nofollow" name="robots"> to the head section of each page which should accomplish my goal. I'd still like to know how to add a robots.txt too though. –  Mike Deck Feb 25 '13 at 15:36
    
Can't you just manually upload a robots.txt over FTP? WordPress doesn't use it anyway. –  Camil Staps Feb 25 '13 at 15:44
add comment

2 Answers 2

up vote 2 down vote accepted

First of all, in order for Wordpress to generate a robots.txt for you you must be using a non-default permalink structure. Make sure you've selected an option in the Settings > Permalinks menu.

Also, if a robots.txt file exists at your root directory it will override the setting in Wordpress. It looks like you already have a robots.txt file and that is the reason the wordpress setting is ignored.

share|improve this answer
    
I don't already have a robots.txt, but adding one does resolve the issue. When I delete the robots.txt off the server I get a 404 when requesting /robots.txt –  Mike Deck Feb 25 '13 at 20:39
1  
lol, your valid url confused me. robots.txt is genrerated by wordpress only if you use pretty permalinks, do you? –  Mark Kaplun Feb 25 '13 at 20:41
    
good call, that was the problem. I hadn't yet set up a permalink structure and was using the default. After I selected the "Post name" option in the Permalinks Settings menu robots.txt started working as expected. –  Mike Deck Feb 26 '13 at 16:07
add comment

You don't need to add robots.txt file to the root of your site. robots.txt file is generated in real time, when you visit http://mysite.com/robots.txt. The function, responsible for creation of this file, is do_robots.

If you wish to add your own directives, just write your hook for robots_txt filter, like this:

add_filter( 'robots_txt', 'wpse8170_my_robots_txt', 10, 2 );
function wpse8170_my_robots_txt( $output, $public ) {
    if ( '0' != $public ) {
        $output .= '
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
';
    }
    return $output;
}
share|improve this answer
    
You say "robots.txt file is generated in real time, when you visit mysite.com/robots.txt";, but that's not happening for me. I'm getting a 404 when visiting that URL. –  Mike Deck Feb 25 '13 at 20:37
    
robots.txt is generated only if you use pretty permalinks. So to make it working go to Settings > Permalinks and select permalink structure you, which you need. –  Eugene Manuilov Feb 26 '13 at 16:11
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.