What is a good robots.txt?

Question

What is the "best" setup for robots.txt?
I'm using the following permalink structure /%category%/%postname%/.

My robots.txt currently looks like this (copied from somewhere a long time ago):

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments

1. I want my comments to be indext. So I can remove this
2. Do I want to disallow indexing categories because of my permalinkstructure?
3. An article can have several tags and be in multiple categories. This may cause duplicates in google. How should I work around this?

Would you change anything else here?

Answer 1

FWIW, trackback URLs issue redirects and have no content, so they won't get indexed.

And at the risk of not answering the question, RE your points 2 and 3:

http://googlewebmastercentral.blogspot.com/2008/09/demystifying-duplicate-content-penalty.html

Put otherwise, I think you're wasting your time worrying about dup content, and your robots.txt should be limited to:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes

Answer 2

Have you looked at Yoast's WordPress SEO plugin? It definitely handles robots.txt issues.

Answer 3

With a little bit of help, this is now mines (not to much different from everyone elses, apparently)

User-agent: *
    Allow: /

Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /cat/
    Disallow: /key/
    Disallow: /*?
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes

User-agent: Mediapartners-Google
    Allow: /

User-agent: Adsbot-Google
    Allow: /

User-agent: Googlebot-Image
    Allow: /

User-agent: Googlebot-Mobile
    Allow: /

#User-agent: ia_archiver-web.archive.org
    #Disallow: /

Sitemap: YOURSITENAME.HERE

Answer 4

FYI, ALWYAS begin your permalink with a number. From experience it speeds up the page because wordpress can quickly differentiate between a page and a post (I also read that somewhere else then tried it..and its true). so http:example.com/%month%/%post%...will be fine

I am just going to copy what I have. A lot of research went into this. It's probably overkill! It does help with google recognizing what the main keywords of your site are as seen in the Google webmasters tool. Hope it helps

User-agent:*
Allow: /
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /cgi-bin/
Sitemap: Url to sitemap1
Sitemap: Url to sitemap2

User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.cgi$
Disallow: /*.wmv$
Disallow: /*.ico$
Disallow: /*.opml$
Disallow: /*.shtml$
Disallow: /*.jpg$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /wp-*
Allow: /wp-content/uploads/ 

# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*

User-agent:  *
Disallow: /about/
Disallow: /contact-us/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-

# disallow archiving site
User-agent: ia_archiver
Disallow: /

# disable duggmirror
User-agent: duggmirror
Disallow: /

User-agent: Googlebot
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*