Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

What is the "best" setup for robots.txt?
I'm using the following permalink structure /%category%/%postname%/.

My robots.txt currently looks like this (copied from somewhere a long time ago):

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments
  1. I want my comments to be indext. So I can remove this
  2. Do I want to disallow indexing categories because of my permalinkstructure?
  3. An article can have several tags and be in multiple categories. This may cause duplicates in google. How should I work around this?

Would you change anything else here?

share|improve this question
    
You might get better answers on the Pro Webmasters Stack Exchange, and then return here to know how to implement these tips with WordPress. –  Jan Fabry Dec 13 '10 at 10:57
    
That was the first thing I did. But since this is WP specific, I deleted my Q and posted it here instead - as this is more wp related.... –  Steven Dec 13 '10 at 11:07
    
PS. Implementing is just putting the robots.txt in my WP directory. –  Steven Dec 13 '10 at 11:11

4 Answers 4

up vote 4 down vote accepted

FWIW, trackback URLs issue redirects and have no content, so they won't get indexed.

And at the risk of not answering the question, RE your points 2 and 3:

http://googlewebmastercentral.blogspot.com/2008/09/demystifying-duplicate-content-penalty.html

Put otherwise, I think you're wasting your time worrying about dup content, and your robots.txt should be limited to:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
share|improve this answer
    
Thanks for that. I will however not index authors. Authors come and go. If I delete one, I'll get a bunch of 404 in Google. I know because I've got several of them now :) –  Steven Dec 15 '10 at 15:40

Have you looked at Yoast's WordPress SEO plugin? It definitely handles robots.txt issues.

share|improve this answer
    
I don't want to use more plugins than I really have to. –  Steven Dec 13 '10 at 12:38
    
Okay, here's the first question: why not? Second question: why not consider WP SEO a "have to" plugin? –  ZaMoose Dec 13 '10 at 14:23
2  
The more plugins I use, the more I loos control over what code is on my site. Therefore I preffere to use my own code. Besides, I fairly confident in my own SEO skills. –  Steven Dec 15 '10 at 15:30

With a little bit of help, this is now mines (not to much different from everyone elses, apparently)

User-agent: *
    Allow: /

Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /cat/
    Disallow: /key/
    Disallow: /*?
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes

User-agent: Mediapartners-Google
    Allow: /

User-agent: Adsbot-Google
    Allow: /

User-agent: Googlebot-Image
    Allow: /

User-agent: Googlebot-Mobile
    Allow: /

#User-agent: ia_archiver-web.archive.org
    #Disallow: /

Sitemap: YOURSITENAME.HERE
share|improve this answer

FYI, ALWYAS begin your permalink with a number. From experience it speeds up the page because wordpress can quickly differentiate between a page and a post (I also read that somewhere else then tried it..and its true). so http:example.com/%month%/%post%...will be fine

I am just going to copy what I have. A lot of research went into this. It's probably overkill! It does help with google recognizing what the main keywords of your site are as seen in the Google webmasters tool. Hope it helps

User-agent:*
Allow: /
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /cgi-bin/
Sitemap: Url to sitemap1
Sitemap: Url to sitemap2

User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.cgi$
Disallow: /*.wmv$
Disallow: /*.ico$
Disallow: /*.opml$
Disallow: /*.shtml$
Disallow: /*.jpg$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /wp-*
Allow: /wp-content/uploads/ 

# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*

User-agent:  *
Disallow: /about/
Disallow: /contact-us/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-

# disallow archiving site
User-agent: ia_archiver
Disallow: /

# disable duggmirror
User-agent: duggmirror
Disallow: /

User-agent: Googlebot
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.