WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems.

1- I get this message : "Comments for FirstName LastName name-last.com/‎ A description for this result is not available because of this site's robots.txt – learn more." When I google my name. I re-checked setting and I haven't checked the "discourage option"

2- I installed WP Better Security and Yoast SEO plugin. Now I have another problem!

When I google my domain RANDOM Web Sniffing Websites come up that exposes my data server and many more thing!

Where did I go wrong ?

PS: My robots.txt is fixed to disallow only Plugin Folder since yesterday (not too long)

share|improve this question
Are your privacy settings correct? – Tom J Nowell Aug 24 '13 at 20:31
by privacy you mean my .htaccess file or my setting inside WordPress ? 1- I haven't touched the .ht file, I only created a robot.txt file 2- and for the setting I un-checked the part that says discourage browsers. – Observer Aug 24 '13 at 20:34
Maybe this question should be migrated to the Webmaster forum? – epilektric Aug 24 '13 at 21:28
Regarding my previous comment - Other than allowing editing of robots.txt I'm not sure Yoast SEO is a cause of the problem and WP Better Security doesn't alter it as far as I'm aware. – epilektric Aug 24 '13 at 21:35

Your robots file as of right now:

User-agent: Googlebot
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.yadfaeq.com

You aren't specifying anything for Googlebot so you should remove it. Additionally, Disallow: could be interpreted as Disallow: / which would block your entire site.

What your robots file should be:

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.yadfaeq.com

Google often caches the robots.txt file so it may take some time for it to check for a new one. I suggest you use Google Webmaster Tools to help control what Google is indexing.

I do not see the changes you said you made to your robots.txt file. Perhaps your server is caching it as well?

share|improve this answer
Thanks for your reply mate. I started implementing the Webmaster Tools and after that my website gets picked by weird Bots and now the first page of Google is all these websites :-( ! The only problem is, for every change I have to wait for a while. I think I messed it up. – Observer Aug 25 '13 at 9:02

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.