Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I was looking for a decent robots.txt solution for my wordpress sites and after some searching I found Yoast's SEO plugin that provides a complete SEO solution including indexation. I set it up it to not index the following pages: search results, login and register pages, all admin pages, date-based archives and tag archives.

However, when I looked at the source of those pages, I couldn't find any meta robots tag except in the login/register page. Is there another way to tell search engines what to index except robots meta and robots.txt? If not, where is the indication to these pages not being indexed?

Thanks in advance

share|improve this question
    
@user1567: Are you saying that yoursite.com/robots.txt does not contain the correct directives? –  MikeSchinkel Jan 24 '11 at 1:57
    
It contains this: User-agent: * Disallow: despite of the settings I have in the plugin. –  user1567 Jan 24 '11 at 7:50

2 Answers 2

If you don’t have a physical robots.txt WP will create one on the fly. The plugin will probably place its instructions there.

share|improve this answer
    
Actually I searched for robots.txt file and couldn't fund one. In addition, once I created robots.txt the plugin did not change this file's contents –  user1567 Jan 23 '11 at 20:09
    
If you create a real file, WordPress isn’t called anymore to build one. The robots.txt build by WP will not be saved as a file, it is just an ad hoc output. –  toscho Jan 23 '11 at 21:07
    
Thanks. So how can I see this output? how do I know these pages are not indexed? –  user1567 Jan 23 '11 at 21:11
    
I see the virtual robots.txt file that wp has created by navigating to mydomain.com/robots.txt. However, it states User-agent: * Disallow: Which is not what I defined in the plugin. It seems like the configuration of the plugin has no effect on the robots.txt file nor the robots meta tags unless I'm doing something wrong –  user1567 Jan 23 '11 at 22:13
    
Sounds like you have excluded search engines in /wp-admin/options-privacy.php. Do you? –  toscho Jan 23 '11 at 22:20

This is a basic robots.txt for WordPress that tells good bots not to index stuff outside your content.

Please be aware many bots and crawlers do not care about robots.txt and do whatever they want.

User-agent: *

Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.