3

I've noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn't have that line, so why does it appear? Is it maybe a strange behaviour of some plugins?

  • 1
    Can you add a link to the exact robots.txt? You could put it in a GitHub Gist if you don't want to link to your own site. – guaka Aug 15 '14 at 21:09
  • gist.github.com/anonymous/2651e36c663cae27da6f I've already tryed modifying it, but the crawl delay line is always added just after the disallow istructions. – LordSilver Aug 16 '14 at 4:11
  • Ok, I've found out it was applied by the server, as I supposed. I solved removing the robots.txt file and using a filter to append my directives to the virtual robots.txt, which actually cannot be modified by the server rules. – LordSilver Aug 16 '14 at 17:22
  • Any idea why the server added this line? Which software etc? Answering your own question might help someone else in the future (possibly even yourself). – guaka Aug 16 '14 at 19:37
  • 1
    I've already answered. I contacted my web hosting technical support and they confirmed that as a general rule they add that line to robots.txt in the shared-hosts plan to keep the bandwidth of the server more stable. – LordSilver Aug 16 '14 at 20:56
2

For those who are using WordPress as CMS for their site, you can bypass your web hosting server rules by simply removing your robots.txt file and instead modifying the virtual one generated by WordPress. You just have to add a filter to the functions.php file of your theme.

Here's the code snippet:

//* Append directives to the virtual robots.txt
add_filter( 'robots_txt', 'robots_mod', 10, 2 );
function robots_mod( $output, $public ) {
    $output .= "Disallow: /wp-content/plugins/\nSitemap: http://www.example.com/sitemap_index.xml";
    return $output;
}

All you have to do is modify the $output with your own directives.

0

For my website, I had the same issue.

Here's my robots.txt file (without the ''crawl delay: 10'')

sitemap: https://baliradar.com/sitemap.xml
User-agent: *
Disallow: /cgi-bin/ 
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /archives/
Disallow: /wp-*
Disallow: /author
Disallow: /comments/feed/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /

I have contacted my web hosting and they confirm me your answer

  • 1
    I solved it by adding a filter to my functions.php file which allows me to append the directives I want to the virtual robot.txt which is generated by WordPress. Your web hosting cannot modify that file since it doesn't exist and it's created by the CMS, which is a good solution actually if you absolutely need your site to be constatly crawled by bots. – LordSilver Jan 5 '16 at 19:55
  • This is something I don't totally understand, WP offers his own robots.txt right ? So no need to add a robots.txt file in the root directory of the website ? Can you explain what kind of filter you added in functions.php file, I would appreciate, thank you. – Didz Jan 7 '16 at 6:03
  • I have made a research on WP forums and apparently many people complained saying that the virtual robots.txr created issue : ''finally got tired of seeing the "file not found" errors in my log and made one robots.txt. myself. Also here's what James Huff a Volunteer Moderator on WP Forum wrote: '' WordPress's virtual Robots.txt is only enabled if you select "I would like to block search engines, but allow normal visitors" in Settings/Privacy or if a plugin enables it via a setting like the one in the sitemap plugin. '' Not sure is a good idea to use WP virtual robots.txt – Didz Jan 7 '16 at 6:10
  • 1
    You can find the filter in my answer. First delete your robots.txt file in your WP root directory. After that, copy and paste the code snippet to your functions.php file (the one inside the WP theme you are currently using). Then modify the $output with your own directives and you are good to go. As regards what you read on WP forums, I've never encountered any problems with "file not found" error for my site. Also, the Privacy tab has been removed from WP settings for a long time now. – LordSilver Jan 7 '16 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.