5

When I try to google search my web address, the follow comes up as the meta description

"A description for this result is not available because of this site's robots.txt – learn more."

The website I have tried to edit the robots.txt (downloaded the WP Robot plugin) and although I have changed it to

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

The robots.txt still appears as

User-agent: *
Disallow: /

I have Yoast SEO, WP Robots, and Salient's Theme Nectar. If anyone might be able to provide me with a solution that would update the robots (or where to go from here) that would be incredibly helpful!

  • 2
    Have you tried to edit the robots.txt file directly (without using plugins)? Have to enabled the search engine visibility? Settings -> Reading -> Search Engine Visibility (Uncheck this option) -> Save – Nikhil Dec 9 '15 at 9:12
  • 3
    Have you tried clearing your cache? You have an X-Powered-By: W3 Total Cache/0.9.3 in your response header for the robots.txt file. – RRikesh Dec 9 '15 at 9:26
  • 1
    Are you using any plugins for caching or security – Pratik bhatt Mar 29 '17 at 4:34
1

Once I experienced the same issue, this is what I did to fix the issue.

Edit the robots.txt file directly (using FTP/SSH),

User-agent: * 
Disallow: /wp-admin/ 
Disallow: /wp-includes/

There are two reasons if the robots files not updated when you edited using a plugin.

  1. File permission.
  2. Some other plugin is reverting the changes.

Also try to update the search engine visibility settings (this might not be an issue, but just have a look).

Settings -> Reading -> Search Engine Visibility (Uncheck this option) -> Save

I hope this helps.

  • As you mentioned edit file directly. how to edit robots.txt directly with FTP, I think its not kind of hard file which you can see and edit via FTP ? – jas Dec 9 '15 at 10:09
  • 1
    Yes, it is (if you are not depending a plugin to create the file). Jut login to FTP, create robots.txt file in your website root add the contents which I mentioned in the answer. – Nikhil Dec 9 '15 at 10:13
0

A WordPress site doesn't need an actual robots.txt file. WordPress itself will intercept requests for robots.txt and respond with its own version, and that's what your plugin is most likely trying to modify.

Delete or rename your actual robots.txt file and try it.

0

Happened to me too, try to "Purge All Caches" on Wordpress settings/hosting service provider.

Then to check, open a browser in "Incognito mode" (chrome, make sure you didn't have any Incognito windows open already, otherwise you'll have to close them all and re-open a new Incognito window), then head over to your www.mydomain.com/robots.txt You should now be able to see the updated content file like expected.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.