0

Can I remove this lines form my robots.txt when using WordPress:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://domain/sitemap_index.xml

-> I want to remove these lines so that I do not show the public that I am using WordPress, I want to make it less obvious to attackers that I am using WordPress so these lines are very obvious, can I delete them for security purposes, without any drawbacks?

2 Answers 2

1

You can disable the robots.txt file by putting this code in an mu-plugin:

remove_action( 'do_robots', 'do_robots' );

However, I don't recommend that. It will make it harder for search engines to index your content, and it won't really do anything to hide the fact that you're using WordPress.

There are a dozen ways that someone can tell you're using WP, and many of them are impractical to hide. For example, viewing the source code of a page will show lots of things that are obviously WP:

<script src='https://wp-develop.test/wp-includes/js/jquery/jquery.js?ver=3.7.0' id='jquery-core-js'></script>
<script src='https://wp-develop.test/wp-includes/js/jquery/jquery-migrate.js?ver=3.4.1' id='jquery-migrate-js'></script>
<link rel="https://api.w.org/" href="https://wp-develop.test/wp-json/" /><link rel="EditURI" type="application/rsd+xml" title="RSD" href="https://wp-develop.test/xmlrpc.php?rsd" />
<meta name="generator" content="WordPress 6.4-alpha-56267-src" />

...

<a class="skip-link screen-reader-text" href="#content">
            Skip to content     </a>

...

<article id="post-1241" class="post-1241 post type-post status-publish format-standard sticky hentry category-sticky">

...

<div class="wp-block-column is-layout-flow wp-block-column-is-layout-flow" style="flex-basis:15%"></div>

They could also visit wp-login.php, xmlrpc.php, /wp-json/, and a bunch of other things. There are some plugins that try to mask some of those, with varying degrees of success, but someone with a little familiarity with WP will almost always be able to tell. Even WP themes tend to have a "look" to them that is sometimes distinguishable.

Hiding the fact that you're running WP is basically "security through obscurity" and "security theater"; it won't really make your site more secure. There are much more important things you should do instead, like enabling two-factor authentication, requiring strong passwords, making sure you're only using reputable plugins/themes/hosting, etc. You can learn more in the Hardening WordPress article.

1
  • Thank you so much for your answer
    – AYNTK
    Aug 18, 2023 at 21:52
1

Note that the robots.txt file for most WordPress sites (which you can see with a URL of www.example.com/?robots=1 ) is normally a virtual response automatically created by WP with the do_robots() function.

The virtual robots.txt file will, by default, contain the directives you specify in your question. You can add to the directives by using the 'robots.txt' filter

add_filter('robots.txt', $new_directives);

That will add the string defined by $new_directives to the virtually generated robots.txt file. You might use this if you wanted to add directives to block AI scanners from scanning your site, for example.

Note also that if you have an actual robots.txt file in your site root, the virtual robots.txt file is not generated or used.

Of course, scanners are not required to use robots.txt, as it is an optional thing for scanner engines to respect. It is easy to make a scanner that will ignore the robots.txt file. But 'responsible' site scanners (like the googles/bings/ducks, etc) will respect the directives in your robots.txt file.

If you want to fully disable the virtually generated robots.txt directives, just place an actual robots.txt file in your site root. And the answer by Ian Dunn is also correct; I'm just providing additional information to consider.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.