3

I have a subfolder WP installation. It serves multiple domains, which are linked to the same folder. The requested domain is received in wp-config.php (from the $_SERVER variable) and used to define WP_SITEURL, WP_HOME and DOMAIN_CURRENT_SITE. So we can open the same both from domain.com and domain.co.uk. I need to add the link to XML sitemap to robots.txt and, obviously, it should be different depending on the domain requested.

There is the do_robots() native WP function which generates robots.txt for multisite and allows changing it dynamically using its robots_txt action from theme's functions.php file or from a plugin. However, it does not seem the case for a single-site installation.

I can call do_robots from theme functions to generate the content and write to the robots.txt file but am not sure where I should hook it.

The question is: how do I have the robots.txt dynamically generated or have the possibility to change its content with hooks from theme functions.php?

2

I just tested the 'robots_txt' filter on a single installation to modify the output of the virtual /robots.txt that WordPress displays and it worked fine for me:

add_filter('robots_txt', 'wpse_248124_robots_txt', 10,  2);

function wpse_248124_robots_txt($output, $public) {

  return 'YOUR DESIRED OUTPUT';
}

What is really happening when you try to reach /robots.txt? Does it display the default robots.txt content or a 404? If you're getting a 404, then you might have Apache or Nginx rules that are not allowing the /robots.txt request to go through PHP. It is very common to have something like this on a nginx configuration:

# Don't log access to /robots.txt
location = /robots.txt {
    access_log    off;
    log_not_found off;
}

You should replace it with something like:

# Don't log access to /robots.txt
location = /robots.txt {
    try_files     $uri $uri/ /index.php?$args;
    access_log    off;
    log_not_found off;
}

You should also check if the WordPress own internal rewrite rules are working correctly using the Rewrite Rules Inspector (or any other method available) by making sure the following rewrite rule exists:

robots\.txt$ index.php?robots=1

If it doesn't, you should try to deactivate plugins, activate a default theme and flush the rewrite rules to check if the rewrite rule comes back, but if you have no time for that, just add this rewrite rule to your .htaccess:

RewriteRule robots\.txt$ index.php?robots=1
  • When I try to reach robots.txt I get 404. I have a multisite installation on this server and it generates robots.txt just fine. The robots_txt filter neither works on my localhost where I do not have any robots restricting rules. I can see there is is_robots check in template loader that calls do_robots if true. The is_robots function checks the is_robots property of $wp_query, which is set to true only if there is the 'robots' query var. I do not see where is this query var set and how we access $wp_query requesting robots.txt at all – Igor Skoldin Dec 3 '16 at 6:36
  • Sorry to insist on the server-side issue, but what do you get if you add a forward slash to the end, like this: /robots.txt/? – Cristiano Baptista Dec 3 '16 at 11:54
  • 1
    Regarding the 'is_robots' var, it goes like this: 1. There is a built-in permalink rule for this: Rule: robots\.txt$ Rewrite: index.php?robots=1 Use 'Rewrite Rules Inspector' plugin to make sure it is really there 2. The rewrite rule query var is captured here by $wp_query 3. is_robots() checks the global $wp_query->is_robots Use pre_get_posts to check is_robots() and do whatever you need – Cristiano Baptista Dec 3 '16 at 12:13
  • Perhaps instead of 'pre_get_posts' you should use template_redirect – Cristiano Baptista Dec 3 '16 at 12:26
  • Thanks for the breaking down. It appeared that I did not have the rewrite rule for robots.txt (perhaps WP does not add it for subfolder installations) so I added it to .htaccess like this RewriteRule robots\.txt$ index.php?robots=1 and now the robots_txt filter works like a charm. The reason why it was not working on my localhost was that I did not have overriding allowed in Apache config. So you were right: that was a server-side issue :) Please edit your answer and add this rewrite rules bit and I will accept it. – Igor Skoldin Dec 4 '16 at 7:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.