0

I can see the robots.txt file here :

http://persian.cc/robots.txt

But Google can't !

I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine and this robots.txt is a virtual one being made by WordPress. Now how to stop wordpress doing that?

OR If I cannot stop wordpress showing that virtual robots.txt file, how can I stop Google looking for it on my website? maybe a .htaccess code or something?

  • 1
    Your robots.txt seems to be empty, right? – j0k Jan 2 '13 at 13:59
  • Just create your own robots.txt and google / bots will take that one. – woony Jan 2 '13 at 15:06
  • Thanks for your comment but I want to know how to stop wordpress creating such a thing dynamically. – Parsa Pejvak Jan 2 '13 at 16:35
2

Two options:

  1. Create a static file robots.txt. Highly recommended.

  2. Filter 'robots_txt':

    add_filter( 'robots_txt', 'wpse_77969_robots' );
    
    function wpse_77969_robots()
    {
        status_header( 204 );
        return '';
    }
    
  • @ParsaPejvak See wordpress.stackexchange.com/questions/73031/… – I recommend a plugin. :) – fuxia Jan 2 '13 at 19:11
  • Thanks for your help. I have added the code, but now it shows the root page. Please check persian.cc/robots.txt | I need it to return a 404 so that google would understand there is no robots.txt – Parsa Pejvak Jan 5 '13 at 10:44
  • The problem is resolved, I just added this: RedirectMatch 404 "robots.txt" in the .htaccess file. Though Thanks – Parsa Pejvak Jan 5 '13 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.