1

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems.

1- I get this message : "Comments for FirstName LastName name-last.com/‎ A description for this result is not available because of this site's robots.txt – learn more." When I google my name. I re-checked setting and I haven't checked the "discourage option"

2- I installed WP Better Security and Yoast SEO plugin. Now I have another problem!

When I google my domain RANDOM Web Sniffing Websites come up that exposes my data server and many more thing!

Where did I go wrong ?

PS: My robots.txt is fixed to disallow only Plugin Folder since yesterday (not too long)

  • Are your privacy settings correct? – Tom J Nowell Aug 24 '13 at 20:31
  • by privacy you mean my .htaccess file or my setting inside WordPress ? 1- I haven't touched the .ht file, I only created a robot.txt file 2- and for the setting I un-checked the part that says discourage browsers. – 0bserver07 Aug 24 '13 at 20:34
  • Maybe this question should be migrated to the Webmaster forum? – epilektric Aug 24 '13 at 21:28
  • Regarding my previous comment - Other than allowing editing of robots.txt I'm not sure Yoast SEO is a cause of the problem and WP Better Security doesn't alter it as far as I'm aware. – epilektric Aug 24 '13 at 21:35
1

Your robots file as of right now:

User-agent: Googlebot
Disallow: 
User-agent: *
Disallow: 
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.yadfaeq.com

You aren't specifying anything for Googlebot so you should remove it. Additionally, Disallow: could be interpreted as Disallow: / which would block your entire site.

What your robots file should be:

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.yadfaeq.com

Google often caches the robots.txt file so it may take some time for it to check for a new one. I suggest you use Google Webmaster Tools to help control what Google is indexing.

I do not see the changes you said you made to your robots.txt file. Perhaps your server is caching it as well?

  • Thanks for your reply mate. I started implementing the Webmaster Tools and after that my website gets picked by weird Bots and now the first page of Google is all these websites :-( ! The only problem is, for every change I have to wait for a while. I think I messed it up. – 0bserver07 Aug 25 '13 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.