Tell me more ×
WordPress Answers is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it?

I use wordpress multisite, this is why I chose to allow blogs.dir

I want only my posts and categories indexed. That's it:)

User-agent: *

Disallow: /cron/
Disallow: /lo/
Disallow: /portfolio.html

Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /wp-login.php

Allow: /wp-content/uploads/
Allow: /wp-content/blogs.dir/

Disallow: /author/
Disallow: /tag/
Disallow: /archives/
Disallow: /2010/*
Disallow: /2011/*
Disallow: /2012/*
Disallow: /about/

Disallow: /trackback
Disallow: */trackback
Disallow: /comments
Disallow: */comments
Disallow: /feed
Disallow: */feed
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.xlsx $
Disallow: /*.doc$
Disallow: /*.pdf$
Disallow: /*.zip$

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

Ty, take care;)

share|improve this question

1 Answer

up vote 0 down vote accepted

The file robots.txt doesn't exclude all the search engines to crawl your site... Most search engines crawl the hole site without taking the robots.txt file in consideration.

I think that the best way is to use meta tags in the theme of the blogs.

Use the next meta tags for post and categories template files:

<meta name="robots" content="index, follow" /> 
<meta name="googlebot" content="index, follow" />

Use this meta tags for the rest of the template files

<meta name="robots" content="noindex, nofollow, noarchive" />
<meta name="googlebot" content="noindex, nofollow, noarchive" />
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.