WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It only takes a minute to sign up.
Sign up to join this community
Google Webmaster Tools can see it, but I can't find it. Any idea on how it's generated or where it is the file structure? I don't see the file in my root. My website is at http://mikewills.me and the robots.txt file URL is http://mikewills.me/robots.txt.
In the end, Google isn't indexing my site and I am trying to figure out how to edit the robots.txt so that indexing is allowed. I have changed the privacy option to allow indexing, but that hasn't updated the robots.txt.
It's dynamically generated by the function do_robots(), which has both an action (do_robotstxt) and a filter (robots_txt). If you create a robots.txt file in your WordPress root, it will (probably) be served up when /robots.txt is requested, otherwise processing will fall back on WordPress.
Your current file looks like this:
User-agent: *
Disallow:
Sitemap: http://mikewills.me/sitemaps/mikewills-me.xml.gz
The first two settings are appropriate for an indexable blog. So, you may just be waiting for Google at this point. Maybe you need to request reconsideration of your site.
User-agent match * (so all robots), the list of pages you cannot access is [empty]." Which means that they can access all pages.
Oct 7, 2010 at 7:25
robots.txt is dynamically created. There is no robots.txt file. Install a robot.txt plugin will solve your problem. Hope this helps
