1

I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages:

"Url blocked by robots.txt." and "Sitemap contains urls which are blocked by robots.txt."

But I have no robots.txt anywhere. It seems my entire site and every link on it is blocked from being crawled by Googlebot. I was using WordPress' Yoast SEO plugin and Cloudflare, but I disabled them and cleared my cache, to no avail. I thought it was Yoast's fault at first.

WordPress' "Search engine visibility" setting has "Discourage search engines from indexing this site" unchecked, like it should. Now I'm wondering if WordPress creates its own virtual robots.txt or something that doesn't show up in the site root.

What gives? Do I need to wait a few days? I never had a robots.txt to begin with!

Improve this question
0

3 Answers 3

Reset to default
1

Similar problem occurred to me, and I got the solution following these:

  • Step 1: Take a backup of your .htaccess file and then remove it (Don't worry, on next refresh WordPress will create one for you)
  • Step 2: If there's no robots.txt exists, create one with blank page
  • Step 3: Resubmit the sitemap to google webmasters tools
  • Step 4: Additionally use "Crawl as google" and add the site again to confirm that things are working fine

Hope within short time your site will be indexed.

Improve this answer
0
0

Do you have WordPress installed in a sub-directory? If yes, then check the robots file in the root folder.

If no, check if search engine indexing is disabled in Settings > Reading.

If you are using any SEO plugins, check the settings there too. Though indexing is enabled by default in both WordPress core setting and plugins setting, but just to make sure check them too.

Improve this answer
0

Now I'm wondering if WordPress creates its own virtual robots.txt or something that doesn't show up in the site root.

WordPress does exactly this, but only when you are "discouraging" search engines.

What do you get if you go to http://example.com/robots.txt? Where example.com is your site, of course.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.