A lot of time since this quesiton and answer were posted. Since then things has changed a lot. The typical recommendation about disallow crawlers to access wp-content/themes, wp-content/plugins, wp-content/cache, wp-includes, and any other directory that contains CSS or js files needed in the site, are no longer valid.
For example, lets talk about Google. Googlebot was rendering websites without CSS and without js, but not actually. Actually Googlebot fecth the full document and checks things like responsiveness, number, location and size of the scripts, etc. So Google doesn't like if you disallow Googlebot to access CSS and js files. That means that you should not disallow wp-content/themes, wp-content/plugins, wp-content/cache and wp-includes because of all those folders can serve CSS and js files.
From my point of view, actually the best robots.txt file is the one created by WordPress by default (the bellow robots.txt is the default since WP 4.0):
User-agent: *
Disallow: /wp-admin/
If you have a cgi-bin folder, it may be good idea to disallow cgi-bin folder:
User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/
And if you use a sitemap, it is a good idea to include a sitemap reference in robots.txt (you still need to manual submit the sitemap to Google and Bing Webmaster Tools, but the reference can be useful to other crawlers):
User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/
Sitemap: http://example.com/sitemap.xml
That is in general. Specific websites may need disallow other folders and files that should be studied in each specific case. For exmaple, you may need or you may want to disallow a specific plugin folder:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/plugins/plugin-folder/
To modify the robots.txt, use robots_txt filter (using a real robots.txt file will make WordPress be no longer able to handle robots.txt) . For example:
add_filter( 'robots_txt', function( $output ) {
$output .= "Disallow: /cgi-bin/\n";
$output .= "Disallow: /wp-content/plugins/plugin-folder-i-want-to-block/\n";
$output .= "\nSitemap: " . site_url( 'sitemap.xml' ) . "\n";
return $output;
});
robots.txtin my WP directory.