Google appears to be ignoring my robots.txt (despite being valid)

How do I noindex, follow a specific category without plugins?


There are several robots.txt generators online. Here is one of my favorites. After you've generated your file, I highly recommend testing it with Google's Webmaster Tools tester.

I've done this for dozens of sites, and never have a problem.

Just remember the following:

  • Most Wordpress sites treat categories as if they're folders/directories. So, to make a rule for the breads category, you may need /topics/breads/, or /category/breads/, or even /cat/breads/ ... depending on your site's Settings.
  • If you continue to have trouble, you can Deny All by default. Then you would have to expressly permit each category/folder/topic/etc. that you want to have indexed.

EDIT Google provides the following recommendation, which have also worked for me:

//To prevent only most web crawlers from indexing a page:
<meta name="robots" content="noindex">

- or -

//To prevent only Google web crawlers from indexing a page:
<meta name="googlebot" content="noindex">

EDIT I recommend you use a category template. This WPBeginner Article explains how to create them, in case you have not done this before. Basically, you'll usually start by copying your "index.php" or "page.php" to a new "category.php" file. Then, make a couple of simple edits.

Depending on how many categories should be ignored, my typical suggestion might be to create a single category.php template, then use an "If" statement like the following in the template:

// noIndex = list of categories that should be ignored by bots
$noIndex = array('cats','rats','bats',...);
// current_category = category of the page currently being prepared
$current_category = get_the_category();
// If the current_category is in the list of those to be ignored...
if ( in_array($current_category, $noIndex) ){
    // add a filter to insert the meta tag
    function ignore_category_indexing() {
        echo '<meta name="robots" content="noindex" />';
    add_action( 'wp_head', 'ignore_category_indexing' );        

Good luck!

P.S. If this works for you, I could use a Vote Up..

  • Thanks for your answer, I used a robots.txt generator already, and Google is ignoring it. I already have a few categories I'm using, however there are a select number of ID's I'd like to noindex if possible – PublicDisplayName Aug 24 '19 at 0:21
  • Sorry! Didn't realize what you were after. I think I have your solution in my latest edit. Add one or both of the recommended meta tags near the top of your <head> section(s) to block bots from select pages. – Mike Baxter Aug 24 '19 at 0:37
  • How do I add this to one category page only? My understanding is the head section of categories are dynamically generated, I have 5 categories with only 1 I want to noindex – PublicDisplayName Aug 24 '19 at 13:40
  • I just re-read your follow-up question and realized you only have one category to exempt from indexing. Based on that, you should probably create a custom category template for that specific category "category-{slug}.php". Then, implement the filter above without the if statement. – Mike Baxter Aug 26 '19 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.