Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

What is the "best" setup for robots.txt?
I'm using the following permalink structure /%category%/%postname%/.

My robots.txt currently looks like this (copied from somewhere a long time ago):

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments
  1. I want my comments to be indexed. So I can remove this
  2. Do I want to disallow indexing categories because of my permalink structure?
  3. An article can have several tags and be in multiple categories. This may cause duplicates in search providers like Google. How should I work around this?

Would you change anything else here?

share|improve this question
    
You might get better answers on the Pro Webmasters Stack Exchange, and then return here to know how to implement these tips with WordPress. –  Jan Fabry Dec 13 '10 at 10:57
    
That was the first thing I did. But since this is WP specific, I deleted my Q and posted it here instead - as this is more wp related.... –  Steven Dec 13 '10 at 11:07
    
PS. Implementing is just putting the robots.txt in my WP directory. –  Steven Dec 13 '10 at 11:11

6 Answers 6

up vote 3 down vote accepted

FWIW, trackback URLs issue redirects and have no content, so they won't get indexed.

And at the risk of not answering the question, RE your points 2 and 3:

http://googlewebmastercentral.blogspot.com/2008/09/demystifying-duplicate-content-penalty.html

Put otherwise, I think you're wasting your time worrying about dup content, and your robots.txt should be limited to:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-content/cache
share|improve this answer
    
Thanks for that. I will however not index authors. Authors come and go. If I delete one, I'll get a bunch of 404 in Google. I know because I've got several of them now :) –  Steven Dec 15 '10 at 15:40
    
It would be better practice to set an authors role to subscriber, instead of deleting all their content and jumping through loops for SEO purposes –  Tom J Nowell Mar 26 at 22:05
    
I think this answer is no longer the best. I've added an answer. –  cybmeta Mar 27 at 17:30

A lot of time since this quesiton and answer were posted. Since then things has changed a lot. The typical recommendation about disallow crawlers to access wp-content/themes, wp-content/plugins, wp-content/cache, wp-includes, and any other directory that contains CSS or js files needed in the site, are no longer valid.

For example, lets talk about Google. Googlebot was rendering websites without CSS and without js, but not actually. Actually Googlebot fecth the full document and checks things like responsiveness, number, location and size of the scripts, etc. So Google doesn't like if you disallow Googlebot to access CSS and js files. That means that you should not disallow wp-content/themes, wp-content/plugins, wp-content/cache and wp-includes because of all those folders can serve CSS and js files.

From my point of view, actually the best robots.txt file is the one created by WordPress by default (the bellow robots.txt is the default since WP 4.0):

User-agent: *
Disallow: /wp-admin/

If you have a cgi-bin folder, it may be good idea to disallow cgi-bin folder:

User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/

And if you use a sitemap, it is a good idea to include a sitemap reference in robots.txt (you still need to manual submit the sitemap to Google and Bing Webmaster Tools, but the reference can be useful to other crawlers):

User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/

Sitemap: http://example.com/sitemap.xml

That is in general. Specific websites may need disallow other folders and files that should be studied in each specific case. For exmaple, you may need or you may want to disallow a specific plugin folder:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/plugins/plugin-folder/

To modify the robots.txt, use robots_txt filter (using a real robots.txt file will make WordPress be no longer able to handle robots.txt) . For example:

add_filter( 'robots_txt', function( $output ) {

    $output .= "Disallow: /cgi-bin/\n";
    $output .= "Disallow: /wp-content/plugins/plugin-folder-i-want-to-block/\n";
    $output .= "\nSitemap: " . site_url( 'sitemap.xml' ) . "\n";

    return $output;

});
share|improve this answer
1  
I like the fact you mentioned Google and wp-content/themes. This was an issue for our sites because with disallowing that our sites didn't pass Google's "Mobile-Friendly Test" anymore and we got dropped from a major news aggregator mobile app for that reason, even though our sites were fully responsive. –  jimihenrik Mar 27 at 18:18

Have you looked at Yoast's WordPress SEO plugin? It definitely handles robots.txt issues.

share|improve this answer
1  
I don't want to use more plugins than I really have to. –  Steven Dec 13 '10 at 12:38
1  
Okay, here's the first question: why not? Second question: why not consider WP SEO a "have to" plugin? –  ZaMoose Dec 13 '10 at 14:23
4  
The more plugins I use, the more I loos control over what code is on my site. Therefore I preffere to use my own code. Besides, I fairly confident in my own SEO skills. –  Steven Dec 15 '10 at 15:30

You should follow Joost de Valk's current approach where very little is blocked in robots.txt, but also understand that each site will have a uniquely appropriate policy that will need to be reviewed and changed over time.

Many of the answers given here previously are dated and will result in SEO self-sabotage since Google checks for "mobile friendliness" now. Today googlebots try to load everything a normal browser does, including fonts, images, JavaScript, and CSS assets from /wp-content, /themes, /plugins, etc. (Morten Rand-Hendriksen recently blogged about this.)

You can use Google's "mobile friendly" site checker to find out if your robots.txt file is sabotaging your site. If you use Google Webmaster Tools you should receive alerts and emailed notices if there is a big problem.

Unless you are careful to make sure no key presentational or interactive assets are being loaded from disallowed folders, this is probably the bare minimum every WordPress install is safe with:

User-agent: *
Disallow: /wp-admin

And don't forget to add a sitemap:

Sitemap: http://yoursite.com/sitemap.xml

Unfortunately this more open policy today recreates the potential for other problems that formerly led people to be more restrictive with robots.txt, such as [plugin and theme developers including indexable pages with links back to their own sites].4 There is nothing to be done about this unless you are able to pore over all third party code with a fine tooth comb and move or remove things you don't want to be indexed.

share|improve this answer

With a little bit of help, this is now mines (not to much different from everyone elses, apparently)

User-agent: *
    Allow: /

Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /cat/
    Disallow: /key/
    Disallow: /*?
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes

User-agent: Mediapartners-Google
    Allow: /

User-agent: Adsbot-Google
    Allow: /

User-agent: Googlebot-Image
    Allow: /

User-agent: Googlebot-Mobile
    Allow: /

#User-agent: ia_archiver-web.archive.org
    #Disallow: /

Sitemap: YOURSITENAME.HERE
share|improve this answer

FYI, ALWYAS begin your permalink with a number. From experience it speeds up the page because WordPress can quickly differentiate between a page and a post (I also read that somewhere else then tried it..and its true). so http:example.com/%month%/%post%...will be fine

I am just going to copy what I have. A lot of research went into this. It's probably overkill! It does help with Google recognizing what the main keywords of your site are as seen in the Google webmasters tool. Hope it helps

User-agent: *
Allow: /
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /cgi-bin/
Sitemap: Url to sitemap1
Sitemap: Url to sitemap2

User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.cgi$
Disallow: /*.wmv$
Disallow: /*.ico$
Disallow: /*.opml$
Disallow: /*.shtml$
Disallow: /*.jpg$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /wp-*
Allow: /wp-content/uploads/ 

# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*

User-agent:  *
Disallow: /about/
Disallow: /contact-us/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-

# disallow archiving site
User-agent: ia_archiver
Disallow: /

# disable duggmirror
User-agent: duggmirror
Disallow: /

User-agent: Googlebot
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.