Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I can see the robots.txt file here :

http://persian.cc/robots.txt

But Google can't !

I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine and this robots.txt is a virtual one being made by WordPress. Now how to stop wordpress doing that?

OR If I cannot stop wordpress showing that virtual robots.txt file, how can I stop Google looking for it on my website? maybe a .htaccess code or something?

share|improve this question
1  
Your robots.txt seems to be empty, right? –  j0k Jan 2 '13 at 13:59
    
Just create your own robots.txt and google / bots will take that one. –  woony Jan 2 '13 at 15:06
    
Thanks for your comment but I want to know how to stop wordpress creating such a thing dynamically. –  Parsa Pejvak Jan 2 '13 at 16:35
add comment

1 Answer

Two options:

  1. Create a static file robots.txt. Highly recommended.

  2. Filter 'robots_txt':

    add_filter( 'robots_txt', 'wpse_77969_robots' );
    
    function wpse_77969_robots()
    {
        status_header( 204 );
        return '';
    }
    
share|improve this answer
    
Thanks toscho, please explain where to add this code –  Parsa Pejvak Jan 2 '13 at 19:09
    
@ParsaPejvak See wordpress.stackexchange.com/questions/73031/… – I recommend a plugin. :) –  toscho Jan 2 '13 at 19:11
    
Thanks for your help. I have added the code, but now it shows the root page. Please check persian.cc/robots.txt | I need it to return a 404 so that google would understand there is no robots.txt –  Parsa Pejvak Jan 5 '13 at 10:44
    
The problem is resolved, I just added this: RedirectMatch 404 "robots.txt" in the .htaccess file. Though Thanks –  Parsa Pejvak Jan 5 '13 at 11:56
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.