Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

Google Webmaster Tools can see it, but I can't find it. Any idea on how it's generated or where it is the file structure? I don't see the file in my root. My website is at http://mikewills.me and the robots.txt file URL is http://mikewills.me/robots.txt.

In the end, Google isn't indexing my site and I am trying to figure out how to edit the robots.txt so that indexing is allowed. I have changed the privacy option to allow indexing, but that hasn't updated the robots.txt.

share|improve this question
    
Site link? robots.txt is unlikely to have anything to do with WP... –  Rarst Oct 6 '10 at 18:05
add comment

2 Answers

up vote 9 down vote accepted

It's dynamically generated by the function do_robots(), which has both an action (do_robotstxt) and a filter (robots_txt). If you create a robots.txt file in your WordPress root, it will (probably) be served up when /robots.txt is requested, otherwise processing will fall back on WordPress.

Your current file looks like this:

User-agent: *
Disallow:

Sitemap: http://mikewills.me/sitemaps/mikewills-me.xml.gz

The first two settings are appropriate for an indexable blog. So, you may just be waiting for Google at this point. Maybe you need to request reconsideration of your site.

share|improve this answer
    
Thanks for the information. I was under this assumption. I was scared to see the Disallow. So that is okay then? –  Mike Wills Oct 6 '10 at 23:37
2  
@Mike: Yes, this is OK. What you say now is: "Hey robots whose User-agent match * (so all robots), the list of pages you cannot access is [empty]." Which means that they can access all pages. –  Jan Fabry Oct 7 '10 at 7:25
add comment

robots.txt is dynamically created. There is no robots.txt file. Install a robot.txt plugin will solve your problem. Hope this helps

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.