Skip to main content
7 votes

Why does do_robots() Allow: /wp-admin/admin-ajax.php by default?

First: have a read-up on robots.txt, if you haven't. Another good reference is this one from Yoast. The sample you posted: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: ...
Pat J's user avatar
  • 11.9k
4 votes
Accepted

Disallow author with robots.txt

If you use Yoast SEO, you can go to Users > All Users, and in the Yoast SEO settings, check Do not allow search engines to show this author's archives in search results. See this link. OR With ...
Pim's user avatar
  • 1,152
3 votes

Why does do_robots() Allow: /wp-admin/admin-ajax.php by default?

the truth is that probably nothing should be blocked in robots.txt by core (this is IIRC was joost's position abot the matter) as wordpress is an open platform and front end content and styling might ...
Mark Kaplun's user avatar
  • 23.4k
2 votes

Robots.txt not updating

I had the same issue and for me, I had to purge the Cloudflare cache.
Vahid's user avatar
  • 179
2 votes

Preventing Search Engines Indexing Pages 2, 3 and More?

If it is because of SEO and the warnings in the Google Search console, these can be ignored. wp / Page2 and so on should still be indexed. The this answer and the article with the answer from google: ...
Axel Metayer's user avatar
2 votes
Accepted

Generating robots.txt dynamically

I just tested the 'robots_txt' filter on a single installation to modify the output of the virtual /robots.txt that WordPress displays and it worked fine for me: add_filter('robots_txt', '...
Cristiano Baptista's user avatar
2 votes
Accepted

How to prevent Google from indexing the /wp-content/ directory?

Here's the thing There are multiple ways to do what you want, from adding meta tags, to passing headers, but because, you tagged your question with robots.txt So i consider it off-topic to discuss any ...
Aditya Agarwal's user avatar
2 votes

How to prevent Google from indexing the /wp-content/ directory?

Using robots.txt Yes you could use a robots.txt file for this, simply add the following into this file: User-agent: * Disallow: /wp-content/ Notice that you can have multiple Disallow directives if ...
Cyclonecode's user avatar
  • 1,164
1 vote

Can I remove these lines form my WordPress robots.txt

Note that the robots.txt file for most WordPress sites (which you can see with a URL of www.example.com/?robots=1 ) is normally a virtual response automatically created by WP with the do_robots() ...
Rick Hellewell's user avatar
1 vote
Accepted

Can I remove these lines form my WordPress robots.txt

You can disable the robots.txt file by putting this code in an mu-plugin: remove_action( 'do_robots', 'do_robots' ); However, I don't recommend that. It will make it harder for search engines to ...
Ian Dunn's user avatar
  • 4,837
1 vote

Set noindex page-comment from Pages 2, 3 and More?

function noindex_comments_pages() { global $cpage; if (!empty($cpage) && $cpage > 1) { echo '<meta name="robots" content="noindex">'; echo "\n&...
Boyzen's user avatar
  • 23
1 vote
Accepted

Where to put robot.txt and sitemap.xml for wordpress installed in its own directory

You can safely place your robot.txt file as well as the sitemap.xml file into your websites root directory and NOT the WordPress directory. When a robot visits your site, the first place it checks ...
Shawn W's user avatar
  • 226
1 vote

How to remove Wordpress Virtual Robots.txt that keeps overriding my root robots.txt?

Okay, after digging into various areas, I noticed I have a security plugin that is blocking these changes. Problem solved by whitelisting, :) in case anyone else has this issue, check your security ...
Rob's user avatar
  • 44
1 vote
Accepted

Old robots.txt file not changing, can't update to the current robots.txt

Just delete file robots.txt. WordPress is generating virtual robots.txt file, and Yoast SEO plugin (and other plugins too) allows you to customize it. But, if you have an actual robots.txt file, it ...
Milan Petrovic's user avatar
1 vote

"No sitemap linked in your robots.txt file"

Well, that message is pretty self explanatory. There is no sitemap linked in your robots.txt file. So you check if you have a sitemap (if not install a plugin that will generate one for you) and add ...
cjbj's user avatar
  • 14.6k

Only top scored, non community-wiki answers of a minimum length are eligible