-1

Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn't there. Instead the message is displayed as below.

"User-agent: * Disallow: /wp-admin/

Sitemap: http://www.mydomain.com/sitemap.xml"

Please help me, I need to find where is robots.txt file located in wordpress? How to make it again crawled by Google? Or if I need to create a new robots.txt file then how to do that? And what should I do of the previous robot.txt file??

Improve this question
5
  • possible duplicate of Add robots.txt to root
    – Mark Kaplun
    Sep 14, 2014 at 4:25
  • that question says "How to Discourage every search engine from indexing his site". But my question is "How to make Google index my site from crawling robots.txt file (which is not accessible at the moment).
    – Gurjyot Singh
    Sep 14, 2014 at 4:31
  • 1
    You don't need robots.txt to make Google crawl your website. Simply remove it if you have any. Just submit your sitemap to Google Webmaster tools. Robots.txt is needed if you want to discourage search engines to crawl part of your website.
    – Robert hue
    Sep 14, 2014 at 5:28
  • Also, if you want to have robots.txt then it must be in root folder with wp-config.php file.
    – Robert hue
    Sep 14, 2014 at 5:28
  • The searched the root folder, with wp-config.php file but I couldn't find robots.txt file. Is this the reason that Google is unable to crawl it?? Thanks a lot for your help but I not much familiar with server file system that's why my question may look like childish.
    – Gurjyot Singh
    Sep 14, 2014 at 12:24

1 Answer 1

Reset to default
0

Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website.

This is not actually a problem (from crawling perspective at least, it might be for excluding things from crawling, which the file primarily used for). The robots.txt is neither mandatory on necessary for your site to be crawled.

I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn't there. Instead the message is displayed as below.

This actually is robots.txt. :) That is what rules in it look like.

If your installation doesn't have physical robots.txt file than WordPress will be serving virtual one, using do_robots() to generate it.

It is hard to say why Google wasn't to access the file (since you seemingly are able to). It might have been temporary issue that will clear itself, otherwise you might want to contact hosting and inquire if something might be interfering with Google bot accessing your site.

Improve this answer
2
  • Maybe you are correct, again checked in Google webmasters and tried to fetch my site and it says that everything is good again and I have again submitted my sitemap but the request is pending. If that went well I'll again make a comment to tell what's the status. Well thanks for your answer @Rarst
    – Gurjyot Singh
    Sep 14, 2014 at 13:54
  • Yes it was temporary error by Google, now everything is working smooth. Thanks a lot for your help @Rarst
    – Gurjyot Singh
    Sep 14, 2014 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.