Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems.

1- I get this message : "Comments for Yad Faeq yadfaeq.com/‎ A description for this result is not available because of this site's robots.txt – learn more." When I google my name. I re-checked setting and I haven't checked the "discourage option"

2- I installed WP Better Security and Yoast SEO plugin. Now I have another problem!

When I google my domain RANDOM Web Sniffing Websites come up that exposes my data server and many more thing!

Where did I go wrong ?

PS: My robots.txt is fixed to disallow only Plugin Folder since yesterday (not too long)

share|improve this question
    
Are your privacy settings correct? –  Tom J Nowell Aug 24 '13 at 20:31
    
by privacy you mean my .htaccess file or my setting inside WordPress ? 1- I haven't touched the .ht file, I only created a robot.txt file 2- and for the setting I un-checked the part that says discourage browsers. –  Observer Aug 24 '13 at 20:34
    
Maybe this question should be migrated to the Webmaster forum? –  epilektric Aug 24 '13 at 21:28
    
Regarding my previous comment - Other than allowing editing of robots.txt I'm not sure Yoast SEO is a cause of the problem and WP Better Security doesn't alter it as far as I'm aware. –  epilektric Aug 24 '13 at 21:35
add comment

1 Answer 1

Your robots file as of right now:

User-agent: Googlebot
Disallow: 
User-agent: *
Disallow: 
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.yadfaeq.com

You aren't specifying anything for Googlebot so you should remove it. Additionally, Disallow: could be interpreted as Disallow: / which would block your entire site.

What your robots file should be:

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.yadfaeq.com

Google often caches the robots.txt file so it may take some time for it to check for a new one. I suggest you use Google Webmaster Tools to help control what Google is indexing.

I do not see the changes you said you made to your robots.txt file. Perhaps your server is caching it as well?

share|improve this answer
    
Thanks for your reply mate. I started implementing the Webmaster Tools and after that my website gets picked by weird Bots and now the first page of Google is all these websites :-( ! The only problem is, for every change I have to wait for a while. I think I messed it up. –  Observer Aug 25 '13 at 9:02
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.