Tag Info

New answers tagged


Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. This is not actually a problem (from crawling perspective at least, it might be for excluding things from crawling, which the file primarily used for). The robots.txt is neither mandatory on necessary for your site to be crawled. I also ...

Top 50 recent answers are included