I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.

I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but http://mysite.com/robots.txt still returns a 404.

Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?

  • I just realized that the "Discourage search engines from indexing this site" setting adds a <meta content="noindex,nofollow" name="robots"> to the head section of each page which should accomplish my goal. I'd still like to know how to add a robots.txt too though.
    – Mike Deck
    Feb 25, 2013 at 15:36
  • Can't you just manually upload a robots.txt over FTP? WordPress doesn't use it anyway.
    – user26607
    Feb 25, 2013 at 15:44

2 Answers 2


First of all, in order for Wordpress to generate a robots.txt for you you must be using a non-default permalink structure. Make sure you've selected an option in the Settings > Permalinks menu.

Also, if a robots.txt file exists at your root directory it will override the setting in Wordpress. It looks like you already have a robots.txt file and that is the reason the wordpress setting is ignored.

  • I don't already have a robots.txt, but adding one does resolve the issue. When I delete the robots.txt off the server I get a 404 when requesting /robots.txt
    – Mike Deck
    Feb 25, 2013 at 20:39
  • 1
    lol, your valid url confused me. robots.txt is genrerated by wordpress only if you use pretty permalinks, do you? Feb 25, 2013 at 20:41
  • good call, that was the problem. I hadn't yet set up a permalink structure and was using the default. After I selected the "Post name" option in the Permalinks Settings menu robots.txt started working as expected.
    – Mike Deck
    Feb 26, 2013 at 16:07

You don't need to add robots.txt file to the root of your site. robots.txt file is generated in real time, when you visit http://mysite.com/robots.txt. The function, responsible for creation of this file, is do_robots.

If you wish to add your own directives, just write your hook for robots_txt filter, like this:

add_filter( 'robots_txt', 'wpse8170_my_robots_txt', 10, 2 );
function wpse8170_my_robots_txt( $output, $public ) {
    if ( '0' != $public ) {
        $output .= '
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
    return $output;
  • You say "robots.txt file is generated in real time, when you visit mysite.com/robots.txt", but that's not happening for me. I'm getting a 404 when visiting that URL.
    – Mike Deck
    Feb 25, 2013 at 20:37
  • robots.txt is generated only if you use pretty permalinks. So to make it working go to Settings > Permalinks and select permalink structure you, which you need. Feb 26, 2013 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.