2

I can see the robots.txt file here :

http://persian.cc/robots.txt

But Google can't !

I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine and this robots.txt is a virtual one being made by WordPress. Now how to stop wordpress doing that?

OR If I cannot stop wordpress showing that virtual robots.txt file, how can I stop Google looking for it on my website? maybe a .htaccess code or something?

4
  • 1
    Your robots.txt seems to be empty, right?
    – j0k
    Jan 2, 2013 at 13:59
  • Just create your own robots.txt and google / bots will take that one.
    – woony
    Jan 2, 2013 at 15:06
  • Thanks for your comment but I want to know how to stop wordpress creating such a thing dynamically. Jan 2, 2013 at 16:35
  • There should be a way to turn off this "feature" entirely. Adding a filter is not an ideal solution, also it only affects the one theme via functions.php and only the one blog. This should be configurable via wp-config.php and/or make this an "opt-in" feature! Making this mandatory is a huge pain. Also it seems to break DuckDuckGo!
    – PJ Brunet
    Mar 15 at 9:14

1 Answer 1

2

Two options:

  1. Create a static file robots.txt. Highly recommended.

  2. Filter 'robots_txt':

    add_filter( 'robots_txt', 'wpse_77969_robots' );
    
    function wpse_77969_robots()
    {
        status_header( 204 );
        return '';
    }
    
4
  • @ParsaPejvak See wordpress.stackexchange.com/questions/73031/… – I recommend a plugin. :)
    – fuxia
    Jan 2, 2013 at 19:11
  • Thanks for your help. I have added the code, but now it shows the root page. Please check persian.cc/robots.txt | I need it to return a 404 so that google would understand there is no robots.txt Jan 5, 2013 at 10:44
  • The problem is resolved, I just added this: RedirectMatch 404 "robots.txt" in the .htaccess file. Though Thanks Jan 5, 2013 at 11:56
  • Is this code still working? It didn't work for me. Also I think Googlebot prefers a 404 or 200, but no guarantees LOL. Why is WordPress generating robots.txt now? What a pain!
    – PJ Brunet
    Mar 15 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.