Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

What is the "best" setup for robots.txt?
I'm using the following permalink structure /%category%/%postname%/.

My robots.txt currently looks like this (copied from somewhere a long time ago):

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments
  1. I want my comments to be indexed. So I can remove this
  2. Do I want to disallow indexing categories because of my permalink structure?
  3. An article can have several tags and be in multiple categories. This may cause duplicates in search providers like Google. How should I work around this?

Would you change anything else here?

share|improve this question
    
You might get better answers on the Pro Webmasters Stack Exchange, and then return here to know how to implement these tips with WordPress. –  Jan Fabry Dec 13 '10 at 10:57
    
That was the first thing I did. But since this is WP specific, I deleted my Q and posted it here instead - as this is more wp related.... –  Steven Dec 13 '10 at 11:07
    
PS. Implementing is just putting the robots.txt in my WP directory. –  Steven Dec 13 '10 at 11:11

5 Answers 5

up vote 4 down vote accepted

FWIW, trackback URLs issue redirects and have no content, so they won't get indexed.

And at the risk of not answering the question, RE your points 2 and 3:

http://googlewebmastercentral.blogspot.com/2008/09/demystifying-duplicate-content-penalty.html

Put otherwise, I think you're wasting your time worrying about dup content, and your robots.txt should be limited to:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
share|improve this answer
    
Thanks for that. I will however not index authors. Authors come and go. If I delete one, I'll get a bunch of 404 in Google. I know because I've got several of them now :) –  Steven Dec 15 '10 at 15:40
    
It would be better practice to set an authors role to subscriber, instead of deleting all their content and jumping through loops for SEO purposes –  Tom J Nowell 8 hours ago

You should follow Joost de Valk's current approach where very little is blocked in robots.txt, but also understand that each site will have a uniquely appropriate policy that will need to be reviewed and changed over time.

Many of the answers given here previously are dated and will result in SEO self-sabotage since Google checks for "mobile friendliness" now. Today googlebots try to load everything a normal browser does, including fonts, images, JavaScript, and CSS assets from /wp-content, /themes, /plugins, etc. (Morten Rand-Hendriksen recently blogged about this.)

You can use Google's "mobile friendly" site checker to find out if your robots.txt file is sabotaging your site. If you use Google Webmaster Tools you should receive alerts and emailed notices if there is a big problem.

Unless you are careful to make sure no key presentational or interactive assets are being loaded from disallowed folders, this is probably the bare minimum every WordPress install is safe with:

User-agent: *
Disallow: /wp-admin

And don't forget to add a sitemap:

Sitemap: http://yoursite.com/sitemap.xml

Unfortunately this more open policy today recreates the potential for other problems that formerly led people to be more restrictive with robots.txt, such as [plugin and theme developers including indexable pages with links back to their own sites].4 There is nothing to be done about this unless you are able to pore over all third party code with a fine tooth comb and move or remove things you don't want to be indexed.

share|improve this answer

FYI, ALWYAS begin your permalink with a number. From experience it speeds up the page because wordpress can quickly differentiate between a page and a post (I also read that somewhere else then tried it..and its true). so http:example.com/%month%/%post%...will be fine

I am just going to copy what I have. A lot of research went into this. It's probably overkill! It does help with google recognizing what the main keywords of your site are as seen in the Google webmasters tool. Hope it helps

User-agent:*
Allow: /
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /cgi-bin/
Sitemap: Url to sitemap1
Sitemap: Url to sitemap2

User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.cgi$
Disallow: /*.wmv$
Disallow: /*.ico$
Disallow: /*.opml$
Disallow: /*.shtml$
Disallow: /*.jpg$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /wp-*
Allow: /wp-content/uploads/ 

# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*

User-agent:  *
Disallow: /about/
Disallow: /contact-us/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-

# disallow archiving site
User-agent: ia_archiver
Disallow: /

# disable duggmirror
User-agent: duggmirror
Disallow: /

User-agent: Googlebot
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
share|improve this answer

With a little bit of help, this is now mines (not to much different from everyone elses, apparently)

User-agent: *
    Allow: /

Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /cat/
    Disallow: /key/
    Disallow: /*?
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes

User-agent: Mediapartners-Google
    Allow: /

User-agent: Adsbot-Google
    Allow: /

User-agent: Googlebot-Image
    Allow: /

User-agent: Googlebot-Mobile
    Allow: /

#User-agent: ia_archiver-web.archive.org
    #Disallow: /

Sitemap: YOURSITENAME.HERE
share|improve this answer

Have you looked at Yoast's WordPress SEO plugin? It definitely handles robots.txt issues.

share|improve this answer
    
I don't want to use more plugins than I really have to. –  Steven Dec 13 '10 at 12:38
1  
Okay, here's the first question: why not? Second question: why not consider WP SEO a "have to" plugin? –  ZaMoose Dec 13 '10 at 14:23
2  
The more plugins I use, the more I loos control over what code is on my site. Therefore I preffere to use my own code. Besides, I fairly confident in my own SEO skills. –  Steven Dec 15 '10 at 15:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.