Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

Do you know how to prevent indexing of pages past the home page in WP? I mean I don't want mysite.com/page/2, mysite.com/page/3 to be indexed.

This is because I use home.php for my theme, so that page/2, page/3 are all the same.

Please give me a hint or a code snippet please, I don't want to add another plugin (robots meta).

share|improve this question
    
@rich@ - Do you want to limit all search engines or is just Google (and Bing) good enough? –  MikeSchinkel Oct 22 '10 at 6:40
    
all search engines, but why does it matter –  RichZenMaster Oct 22 '10 at 6:45
    
@rich - Well you can easily "sniff" out a handful of search engines and perform a redirect but it's a lot harder to sniff out every search engine there is, just see this list: thesearchenginelist.com It's like finding a needle in a haystack vs. proving there isn't one. You can use the meta robots trick but not all spyders behave. What's your reason to do this? I'm not questioning you, it's just easier to give you a solution that meets your objectives if I know what your objectives are. –  MikeSchinkel Oct 22 '10 at 6:56
    
To prevent duplicate content penalty. If there are many pages with same content as homepage, it may be an issue. –  RichZenMaster Oct 22 '10 at 7:26

3 Answers 3

How exactly are you setting up your home page? I think the problem is with it having unwanted pagination in first place and not that pagination being indexed.

In general robots.txt file is good way to prevent indexing in bulk. I think it would be following directive in your case (please test it so it doesn't affec pagination in other places):

User-agent: *
Disallow: /page/
share|improve this answer
    
unwanted pagination? I don't use any plugin or template tags to paginate as I don't need it. But I still see mysite.com/page/2 in google's index? –  RichZenMaster Oct 22 '10 at 13:09
    
Also, I think that except sub pages of site, there are no pages that have /page/ permalinks, so the above thing will work, but Mike says above that, "not all spyders behave", so I am looking for a wp solution that redirects or prevents indexing. Until then, I'll use this one. Thanks. –  RichZenMaster Oct 22 '10 at 13:11
1  
@Rarst: WordPress tries to handle robots.txt itself, so if you want to play nice with it you should hook into the do_robotstxt or robots_txt hooks. –  Jan Fabry Nov 2 '10 at 12:23
    
@Jan Fabry I know that now, but I used robots.txt long time before that at my blog and never had any issues. If normal robots.txt is present WP just stays away from it. –  Rarst Nov 2 '10 at 14:14
1  
@Jan Fabry setting blog to private also adds nofolow and noindex meta tags to all pages, so search engines are still blocked. –  Rarst Nov 2 '10 at 19:48

If you're trying to prevent duplicate content you should look at the root of the problem. You state that your homepage is using a home.php template, does this include some static text which you're passing to all others pages using the home template? If this is the case either remove it or create a unique home template, which in all honestly should be home.php.

if for whatever reason you want to keep the pages that display the same content as your homepage but using a different URL you can always resort to canonicals.

If you replace the content of your header.php with the following you can specify different headers, one that'll include a canonical that and those that wont.

<?php
if (is_page('1')){
    <?php include(TEMPLATEPATH.'/header1.php'); ?>
}
elseif (is_page('2')){
    <?php include(TEMPLATEPATH.'/header2.php'); ?>
}
else {
    <?php include(TEMPLATEPATH.'/headerdefault.php'); ?>
}
?>

And than you just make sure that you include the canonical which refers to your homepage

<link rel="canonical" href="http://www.yourdomain.com/" />

This will tell Google what is the appropriate URL to the content it's viewing without resorting to using a Plugin.

Either way this all seems a bit weird and I fear that I am just misunderstanding your request as it doesn't seem to make sense. Are you aware of how duplicate content works? Or is it me who should be heading back to bed.

I fail to see the purpose to willingly creating new pages that contain the same content and than looking for a solution to prevent duplicate content.

share|improve this answer

I think the robots meta tags are what need adjusting. You want the spiders to go to page 2, and follow the links to your articles, but you don't want it actually indexing that page (since it will change). So in your header.php, find the "robots" meta tag, and change it to the following:

<meta name="robots" content="follow, <?php echo (get_query_var('paged')==1)?'index':'noindex'?>" />

Using a blanket robots.txt will unfortunately cause the spider to not follow the links, and not find the articles that are on the other pages.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.