7
votes
Why does do_robots() Allow: /wp-admin/admin-ajax.php by default?
First: have a read-up on robots.txt, if you haven't. Another good reference is this one from Yoast.
The sample you posted:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: ...
- 12.3k
4
votes
Accepted
Disallow author with robots.txt
If you use Yoast SEO, you can go to Users > All Users, and in the Yoast SEO settings, check Do not allow search engines to show this author's archives in search results. See this link.
OR
With ...
- 1,152
3
votes
Why does do_robots() Allow: /wp-admin/admin-ajax.php by default?
the truth is that probably nothing should be blocked in robots.txt by core (this is IIRC was joost's position abot the matter) as wordpress is an open platform and front end content and styling might ...
- 23.7k
2
votes
Accepted
Generating robots.txt dynamically
I just tested the 'robots_txt' filter on a single installation to modify the output of the virtual /robots.txt that WordPress displays and it worked fine for me:
add_filter('robots_txt', '...
2
votes
2
votes
Preventing Search Engines Indexing Pages 2, 3 and More?
If it is because of SEO and the warnings in the Google Search console, these can be ignored. wp / Page2 and so on should still be indexed. The this answer and the article with the answer from google:
...
- 21
2
votes
Accepted
How to prevent Google from indexing the /wp-content/ directory?
Here's the thing
There are multiple ways to do what you want, from adding meta tags, to passing headers, but because, you tagged your question with robots.txt So i consider it off-topic to discuss any ...
- 266
2
votes
How to prevent Google from indexing the /wp-content/ directory?
Using robots.txt
Yes you could use a robots.txt file for this, simply add the following into this file:
User-agent: *
Disallow: /wp-content/
Notice that you can have multiple Disallow directives if ...
- 1,164
1
vote
Can I remove these lines form my WordPress robots.txt
Note that the robots.txt file for most WordPress sites (which you can see with a URL of www.example.com/?robots=1 ) is normally a virtual response automatically created by WP with the do_robots() ...
- 7,106
1
vote
Accepted
Can I remove these lines form my WordPress robots.txt
You can disable the robots.txt file by putting this code in an mu-plugin:
remove_action( 'do_robots', 'do_robots' );
However, I don't recommend that. It will make it harder for search engines to ...
- 4,857
1
vote
Set noindex page-comment from Pages 2, 3 and More?
function noindex_comments_pages()
{
global $cpage;
if (!empty($cpage) && $cpage > 1) {
echo '<meta name="robots" content="noindex">';
echo "\n&...
- 23
1
vote
Accepted
Where to put robot.txt and sitemap.xml for wordpress installed in its own directory
You can safely place your robot.txt file as well as the sitemap.xml file into your websites root directory and NOT the WordPress directory.
When a robot visits your site, the first place it checks ...
- 226
1
vote
How to remove Wordpress Virtual Robots.txt that keeps overriding my root robots.txt?
Okay, after digging into various areas, I noticed I have a security plugin that is blocking these changes. Problem solved by whitelisting, :) in case anyone else has this issue, check your security ...
- 44
1
vote
Accepted
Old robots.txt file not changing, can't update to the current robots.txt
Just delete file robots.txt. WordPress is generating virtual robots.txt file, and Yoast SEO plugin (and other plugins too) allows you to customize it. But, if you have an actual robots.txt file, it ...
- 1,864
Only top scored, non community-wiki answers of a minimum length are eligible