0

My Wordpress does not generate a robots.txt file. I have no idea how to solve this and I hope somebody has met with the same problem and knows of a solution. Thanks in advance! Bye Ward

5
  • At first glance you can't easily turn it off inside WordPress, although you could replace the do_robots hook handler with e.g. wp_die( '', 404 ); I suppose.
    – Rup
    Sep 29, 2023 at 13:15
  • Any chance you're blocking it in your .htaccess or equivalent web server config?
    – Rup
    Sep 29, 2023 at 13:15
  • Or do you have any plugins e.g. SEO plugins that might override robots.txt too?
    – Rup
    Sep 29, 2023 at 13:18
  • 2
    SEO plugins such as yoast turning it off are the usual cause of this, there isn't enough information here to answer the question though, what debugging steps have you taken to diagnose this? What happens when you visit /robots.txt? Are you trying to find a physical robots.txt in your CPanel/FTP? ( WP doesn't create or use a physical file, it generates it dynamically on request )
    – Tom J Nowell
    Sep 29, 2023 at 14:57
  • Thanks for the feedback! After some lengthy back and fro with the host, it proved to be a faulty setting in the nginx config file, blocking anything happening with the robots.txt So happily, problem solved! Thanks again for the answers and tips! Bye Ward Oct 2, 2023 at 8:17

2 Answers 2

0

The WP robots.txt file is a virtual file, created on demand. You can see the created file by entering a URL like this https://www.example.com?robots=1 .

The virtual file is generated and used if there is not an actual robots.txt file. If you have an actual one, the virtual file is not generated or supplied on request.

The default directives in the virtual robots.txt file are:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml

The sitemap directive is added if you have a sitemap file.

You can add statements to the generated virtual robots.txt file with the robots.txt filter:

add_filter('robots_txt', 'my_robots_commands', 99, 2); // filter to add robots

function my_robots_commands() {
  $output = "* another command"; // add your additional directives as needed
  return $output;
}

I use that filter to add additional commands to block the various AI site scanners from scanning my sites for AI use.

I wrote a plugin that does that chatbot blocking, but it's been in the plugin 'approve' queue for 2 months now (along with 1200+ other plugins awaiting review). I ended up adding the code to another existing plugin of mine. But you can place the code in your child theme's functions file.

Added

If the URL https://www.example.com?robots=1 does not show any content, it's possible that your theme might have disabled the 'robots.txt' filter. A search for 'robots.txt' in your theme files (or even your plugin files) might indicate that scenario.

1
  • Thanks for the feedback! After some lengthy back and fro with the host, it proved to be a faulty setting in the nginx config file, blocking anything happening with the robots.txt So happily, problem solved! Thanks again for the answers and tips! Bye Ward Oct 2, 2023 at 8:17
-1

Thanks for the feedback! After some lengthy back and fro with the host, it proved to be a faulty setting in the nginx config file, blocking anything happening with the robots.txt So happily, problem solved! Thanks again for the answers and tips! Bye Ward

1
  • could you tell me what you applied because I have the same problem, I use Nginx. Mar 1 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.