Morning all! I've just built a website for a client, using WordPress and the Yoast SEO plugin (free version).


Whilst they're really happy with the website, I've noticed an issue when trying to submit it to search engines, such as Google, for indexing. It is continually being blocked, with an error like this:

Crawl allowed?
No: blocked by robots.txt

I'm not 100% sure why this is the case. I've checked the robots.txt file and there's nothing in it that suggests it would block bots. I have also been through the WordPress installation to ensure that there are no settings (as far as I can see) which are actively blocking search engine indexing.

Is there anything you can suggest as to why this might be the case, and what I can do to resolve it? I of course want to be able to submit the website (sitemaps and URLs) for indexing and I have never come across this issue before. I have also followed advice in the following StackExchange links, but to no avail:

Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?

Google saying is blocked by robots.txt without robots.txt on my website

I am happy to provide additional information if that will help.

Thanks in advance!

  • 1
    I'd try removing the meta robots tag just in case that's the problem, but I think you'll do better asking in a Google support forum. You could try adding a robots.txt that explicitly allows all URLs on the site too, and / or a sitemap.xml (Yoast does these I think?)
    – Rup
    Nov 22, 2021 at 13:51
  • 1
    Thanks @rup. Having created a blank robots.txt file and removed the failed sitemaps from the search console, it has started working! Seems the original robots.txt file that I had (which contained disallow: /) 'tripped' the Google system, so I just needed to delete them and start again.
    – James
    Nov 22, 2021 at 15:02


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.