Setting up robots.txt with Yoast's Seo plugin

Question

I was looking for a decent robots.txt solution for my wordpress sites and after some searching I found Yoast's SEO plugin that provides a complete SEO solution including indexation. I set it up it to not index the following pages: search results, login and register pages, all admin pages, date-based archives and tag archives.

However, when I looked at the source of those pages, I couldn't find any meta robots tag except in the login/register page. Is there another way to tell search engines what to index except robots meta and robots.txt? If not, where is the indication to these pages not being indexed?

Thanks in advance

Answer 1

Writing a robots.txt is an easy process. Follow these simple steps:

Open Notepad, Microsoft Word or any text editor and save the file as 'robots,' all lowercase, making sure to choose .txt as the file type extension (in Word, choose 'Plain Text' ).

Next, add the following two lines of text to your file:

User-agent: *
Disallow:

'User-agent' is another word for robots or search engine spiders. The asterisk (*) denotes that this line applies to all of the spiders. Here, there is no file or folder listed in the Disallow line, implying that every directory on your site may be accessed. This is a basic robots text file.

Blocking the search engine spiders from your whole site is also one of the robots.txt options. To do this, add these two lines to the file:

User-agent: *
Disallow: /

If you'd like to block the spiders from certain areas of your site, your robots.txt might look something like this:

User-agent: *
Disallow: /database/
Disallow: /scripts/

The above three lines tells all robots that they are not allowed to access anything in the database and scripts directories or sub-directories. Keep in mind that only one file or folder can be used per Disallow line. You may add as many Disallow lines as you need. Source: thesearchguru

Answer 2

If you don’t have a physical robots.txt WP will create one on the fly. The plugin will probably place its instructions there.

Answer 3

This is a basic robots.txt for WordPress that tells good bots not to index stuff outside your content.

Please be aware many bots and crawlers do not care about robots.txt and do whatever they want.

User-agent: *

Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-