Google Webmaster Tools can see it, but I can't find it. Any idea on how it's generated or where it is the file structure? I don't see the file in my root. My website is at http://mikewills.me and the robots.txt file URL is http://mikewills.me/robots.txt.

In the end, Google isn't indexing my site and I am trying to figure out how to edit the robots.txt so that indexing is allowed. I have changed the privacy option to allow indexing, but that hasn't updated the robots.txt.

  • Site link? robots.txt is unlikely to have anything to do with WP...
    – Rarst
    Oct 6, 2010 at 18:05

2 Answers 2


It's dynamically generated by the function do_robots(), which has both an action (do_robotstxt) and a filter (robots_txt). If you create a robots.txt file in your WordPress root, it will (probably) be served up when /robots.txt is requested, otherwise processing will fall back on WordPress.

Your current file looks like this:

User-agent: *

Sitemap: http://mikewills.me/sitemaps/mikewills-me.xml.gz

The first two settings are appropriate for an indexable blog. So, you may just be waiting for Google at this point. Maybe you need to request reconsideration of your site.

  • Thanks for the information. I was under this assumption. I was scared to see the Disallow. So that is okay then?
    – Mike Wills
    Oct 6, 2010 at 23:37
  • 2
    @Mike: Yes, this is OK. What you say now is: "Hey robots whose User-agent match * (so all robots), the list of pages you cannot access is [empty]." Which means that they can access all pages.
    – Jan Fabry
    Oct 7, 2010 at 7:25
  • Adding robots.txt to the root does not seem to work, it still serves the WordPress generated one. Is there no way to add or modify the robots.txt without installing a plug-in?
    – Nick
    Oct 8, 2016 at 7:53

robots.txt is dynamically created. There is no robots.txt file. Install a robot.txt plugin will solve your problem. Hope this helps

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.