Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it?

I use wordpress multisite, this is why I chose to allow blogs.dir

I want only my posts and categories indexed. That's it:)

User-agent: *

Disallow: /cron/
Disallow: /lo/
Disallow: /portfolio.html

Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /wp-login.php

Allow: /wp-content/uploads/
Allow: /wp-content/blogs.dir/

Disallow: /author/
Disallow: /tag/
Disallow: /archives/
Disallow: /2010/*
Disallow: /2011/*
Disallow: /2012/*
Disallow: /about/

Disallow: /trackback
Disallow: */trackback
Disallow: /comments
Disallow: */comments
Disallow: /feed
Disallow: */feed
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.xlsx $
Disallow: /*.doc$
Disallow: /*.pdf$
Disallow: /*.zip$

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

Ty, take care;)

share|improve this question

2 Answers 2

up vote 0 down vote accepted

The file robots.txt doesn't exclude all the search engines to crawl your site... Most search engines crawl the hole site without taking the robots.txt file in consideration.

I think that the best way is to use meta tags in the theme of the blogs.

Use the next meta tags for post and categories template files:

<meta name="robots" content="index, follow" /> 
<meta name="googlebot" content="index, follow" />

Use this meta tags for the rest of the template files

<meta name="robots" content="noindex, nofollow, noarchive" />
<meta name="googlebot" content="noindex, nofollow, noarchive" />
share|improve this answer

As for robots.txt file. This will work with major search engines (Google, Bing/Yahoo, Yandex, Baidu).

User-agent: *
Disallow: *.php
Disallow: *.js
Disallow: *.inc
Disallow: *.css
Disallow: *.gz
Disallow: *.wmv
Disallow: *.cgi
Disallow: *.xhtml
Disallow: *.xlsx
Disallow: *.doc
Disallow: *.pdf
Disallow: *.zip
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /author/
Disallow: /tag/
Disallow: /archives/
Disallow: /2010/*
Disallow: /2011/*
Disallow: /2012/*
Disallow: /about/
Disallow: /trackback
Disallow: */trackback
Disallow: /comments
Disallow: */comments
Disallow: /feed
Disallow: */feed
Allow: /wp-content/uploads/
Allow: /wp-content/blogs.dir/
Sitemap: http://domain.com/post_sitemap.xml
Crawl-delay: 4

Make sure you include a sitemap that only has posts. Yoast's Plugin can do this.

Finally (if you don't have a SEO plugin) add the following to your page template:

<meta name="robots" content="noindex, nofollow" />

And this to your post template

<meta name="robots" content="noindex, nofollow" />
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.