1

Suppose this is my URL: https://www.mrskt.com/sitemap_index.xml

I want to only allow specific user agents. I mean only search bots like Google, Bing, Ahrefs, etc. can access that URL.

No other user agent can access that URL.

How can I do that?

1 Answer 1

1

There are at aleast two ways you can block other user agents and allow only a few. This can be done by editing the .htaccess file in the root directory of your WordPress website.

Option 1

Using mod_rewrite, add the code below at the top of your .htaccess file.

<IfModule mod_rewrite.c>

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} !(Google|Bing)
RewriteRule ^(sitemap_index\.xml)$ - [F,L]

</IfModule>

Option 2

This option uses the mod_setenvif module with SetEnvIfNoCase block incase mod_rewrite isn't available. Add the code below at the top of your .htaccess file.

<IfModule mod_setenvif.c>

SetEnvIfNoCase User-Agent .*google.* allowed_user_agents
SetEnvIfNoCase User-Agent .*bing.* allowed_user_agents

Order Allow,Deny
<FilesMatch "^(sitemap_index\.xml)$">
Deny from all
Allow from env=allowed_user_agents
</FilesMatch>

</IfModule>

You could also use robots.txt file, but I haven't tested this one yet.

7
  • The OP is wanting to restrict access to a specific URL. Both your examples restrict access to the entire site.
    – MrWhite
    Aug 24, 2021 at 12:11
  • The code in my answer allows only specific user agents and blocks the rest as the OP is asking.
    – Rodgath
    Aug 24, 2021 at 13:16
  • 1
    ... and to a specific URL, namely /sitemap_index.xml. (It doesn't make sense to restrict an entire site to "search bots" only.)
    – MrWhite
    Aug 24, 2021 at 15:22
  • Hi but i want this only on specific url (mrskt.com/sitemap_index.xml) not the entire site, Please tell...
    – Mr SKT
    Aug 25, 2021 at 8:18
  • 1
    @Rodgath Now, It's Working Perfectly
    – Mr SKT
    Sep 20, 2021 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.