Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows

User-agent: * Disallow:

switching site visibility merely adds/removes a trailing slash ie "Disallow: /"

i have 2 related plugins "all in one seo tools" and "google xml sitemaps" but i disabled both of these to no avail

any ideas anyone .... i know how to override this with a manual sitemap but i'm trying to find out whether this is WordPress intended behavior and if so why

share|improve this question
    
I think I found the answer to this. Apparently Disallow: with no trailing slash is the same as Allow All .... however google doesn't fully comply with this. There is a very good discussion here wordpress.org/support/topic/… –  byronyasgur Aug 8 '11 at 20:03

1 Answer 1

up vote 1 down vote accepted

Disallow with no trailing slash is the same as "allow all" - see here - nice find!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.