Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages:

"Url blocked by robots.txt." and "Sitemap contains urls which are blocked by robots.txt."

But I have no robots.txt anywhere. It seems my entire site and every link on it is blocked from being crawled by Googlebot. I was using WordPress' Yoast SEO plugin and Cloudflare, but I disabled them and cleared my cache, to no avail. I thought it was Yoast's fault at first.

WordPress' "Search engine visibility" setting has "Discourage search engines from indexing this site" unchecked, like it should. Now I'm wondering if WordPress creates its own virtual robots.txt or something that doesn't show up in the site root.

What gives? Do I need to wait a few days? I never had a robots.txt to begin with!

share|improve this question

3 Answers 3

Similar problem occurred to me, and I got the solution following these:

  • Step 1: Take a backup of your .htaccess file and then remove it (Don't worry, on next refresh WordPress will create one for you)
  • Step 2: If there's no robots.txt exists, create one with blank page
  • Step 3: Resubmit the sitemap to google webmasters tools
  • Step 4: Additionally use "Crawl as google" and add the site again to confirm that things are working fine

Hope within short time your site will be indexed.

share|improve this answer

Now I'm wondering if WordPress creates its own virtual robots.txt or something that doesn't show up in the site root.

WordPress does do exactly this, but only when you are "discouraging" search engines.

What do you get if you go to http://example.com/robots.txt ? Where "example.com" is your site, of course.

share|improve this answer

Do you have WordPress installed in a sub-directory? If yes, then check the robots file in the root folder.

If no, check if search engine indexing is disabled in Settings > Reading.

If you are using any SEO plugins, check the settings there too. Though indexing is enabled by default in both WordPress core setting and plugins setting, but just to make sure check them too.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.