0

I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows

User-agent: * Disallow:

switching site visibility merely adds/removes a trailing slash ie "Disallow: /"

i have 2 related plugins "all in one seo tools" and "google xml sitemaps" but i disabled both of these to no avail

any ideas anyone .... i know how to override this with a manual sitemap but i'm trying to find out whether this is WordPress intended behavior and if so why

|improve this question
  • I think I found the answer to this. Apparently Disallow: with no trailing slash is the same as Allow All .... however google doesn't fully comply with this. There is a very good discussion here wordpress.org/support/topic/… – byronyasgur Aug 8 '11 at 20:03
1

Disallow with no trailing slash is the same as "allow all" - see here - nice find!

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.