7

When I try to google search my web address, the follow comes up as the meta description

"A description for this result is not available because of this site's robots.txt – learn more."

The website I have tried to edit the robots.txt (downloaded the WP Robot plugin) and although I have changed it to 

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

The robots.txt still appears as 

User-agent: *
Disallow: /

I have Yoast SEO, WP Robots, and Salient's Theme Nectar. If anyone might be able to provide me with a solution that would update the robots (or where to go from here) that would be incredibly helpful!

Improve this question
3
  • 2
    Have you tried to edit the robots.txt file directly (without using plugins)? Have to enabled the search engine visibility? Settings -> Reading -> Search Engine Visibility (Uncheck this option) -> Save
    – Nikhil
    Commented Dec 9, 2015 at 9:12
  • 3
    Have you tried clearing your cache? You have an X-Powered-By: W3 Total Cache/0.9.3 in your response header for the robots.txt file.
    – RRikesh
    Commented Dec 9, 2015 at 9:26
  • 1
    Are you using any plugins for caching or security
    – Pratik bhatt
    Commented Mar 29, 2017 at 4:34

4 Answers 4

Reset to default
3

Once I experienced the same issue, this is what I did to fix the issue.

Edit the robots.txt file directly (using FTP/SSH),

User-agent: * 
Disallow: /wp-admin/ 
Disallow: /wp-includes/

There are two reasons if the robots files not updated when you edited using a plugin.

  1. File permission.
  2. Some other plugin is reverting the changes.

Also try to update the search engine visibility settings (this might not be an issue, but just have a look).

Settings -> Reading -> Search Engine Visibility (Uncheck this option) -> Save

I hope this helps.

Improve this answer
2
  • As you mentioned edit file directly. how to edit robots.txt directly with FTP, I think its not kind of hard file which you can see and edit via FTP ?
    – jas
    Commented Dec 9, 2015 at 10:09
  • 1
    Yes, it is (if you are not depending a plugin to create the file). Jut login to FTP, create robots.txt file in your website root add the contents which I mentioned in the answer.
    – Nikhil
    Commented Dec 9, 2015 at 10:13
2

I had the same issue and for me, I had to purge the Cloudflare cache.

Improve this answer
0

A WordPress site doesn't need an actual robots.txt file. WordPress itself will intercept requests for robots.txt and respond with its own version, and that's what your plugin is most likely trying to modify.

Delete or rename your actual robots.txt file and try it.

Improve this answer
0

Happened to me too, try to "Purge All Caches" on Wordpress settings/hosting service provider.

Then to check, open a browser in "Incognito mode" (chrome, make sure you didn't have any Incognito windows open already, otherwise you'll have to close them all and re-open a new Incognito window), then head over to your www.mydomain.com/robots.txt You should now be able to see the updated content file like expected.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.