1

I have a WP multisite installed. I am integrated with Ezoic and Cloudflare. I had implemented firewall rules. However, I have an issue blocking bots that have X-Middleton in the User-Agent, because my origin server allows it, for detecting real user IPs.

The X-Middleton is appended when the bots pass through Ezoic that acts as a reverse proxy.  I had inserted a rule in the .htaccess but it is not working, because the origin server allows every request that has X-Middleton.

The rule is as follows:

# BLOCK BOTS
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(ADmantX|Proximic|Barkrowler).*$ [NC]
RewriteRule .* - [F,L]

It is possible a rule that blocks by user agent if the "bad-bot" has X-Middleton in the User-Agent?

2
  • Not sure that I follow? Have you tried modifying that rule for "X-Middleton"?
    – MrWhite
    Mar 2 at 23:33
  • Thanks Mr White I don't know how.
    – Irene
    Mar 3 at 14:58
0

Try it like this:

RewriteCond %{HTTP_USER_AGENT} (ADmantX|Proximic|Barkrowler|X-Middleton) [NC]
RewriteRule ^ - [F]

This will block any request where the User-Agent string contains either ADmantX, Proximic, Barkrowler or X-Middleton. The NC flag makes this a case-insensitive match - whether this is strictly required or not in this example I don't know, but generally this should be a case-sensitive match, since User-Agents (even from bad bots) is usually consistent with regards to case.

The regex prefix ^.* and suffix .*$ are superfluous.

The regex pattern (A|B|C|D) is called alternation. Essentially means A or B or C or D will match.

The RewriteRule pattern .* can be simplified to ^ - which is also marginally more efficient, since you don't need to actually match anything here, just to be successful.

The L flag is not required when the F flag is used - it is implied.


UPDATE:

It appears that X-Middleton (or rather X-Middleton/1) is appended to all User-Agent strings that reach your site, as they pass through the Ezoic reverse proxy. So, simply blocking based on the presence of this string in the User-Agent header (as above) is not going to work, since it will block all requests!

If X-Middleton is simply appended to the UA string and no further processing occurs then you could theoretically block the request when X-Middleton appears twice (or more times) in the UA string in order to block any request where X-Middleton occurs in the original request.

To handle this situation you would create an additional rule. For example:

RewriteCond %{HTTP_USER_AGENT} (X-Middleton).*\1 [NC]
RewriteRule ^ - [F]

\1 is an internal backreference that matches the first matched subpattern, ie. "X-Middleton". So, it is only successful when the string "X-Middleton" occurs at least twice, separated by any number of characters (or none).

The above will block blah X-Middleton blah X-Middleton/1, but not blah blah X-Middleton/1 (case-insensitive match).

However, I would want to see an example access log entry (User-Agent string) of such a request before going live with this. It shouldn't block real user requests, but it might not block fake requests either. If you don't have an actual fake request, then you can mock one up by customising the User-Agent string your browser sends (in Chrome's "Inspector" > Customize Menu > More tools > Network conditions OR install a User-Agent switcher plugin) OR use CURL to make the request, eg. curl -A "<custom-user-agent>" <siteurl> - where you'd set <custom-user-agent> to blah X-Middleton blah or something.

I would also be interested to see the complete list of HTTP request headers that are reaching your application server, as there may be a better way to solve this. (I find it unusual that an intermediate proxy would modify the User-Agent, without also providing the original value. Although, maybe there are no additional options?)

16
  • Very didactic, thanks. However I have a question. All the requests that pass through Ezoic have the X-Middleton in the user-agent. This rule is not going to block all the real users?
    – Irene
    Mar 3 at 16:21
  • Sorry maybe I didn't explain correctly. All the requests that pass through Ezoic append the X-Middleton in the user-agent good bots, bad bots and real users.
    – Irene
    Mar 3 at 16:30
  • Mmmm, yeah, I wondered about that after posting my answer. If all requests contain X-Middleton in the User-Agent then you obviously can't block simply based on the presence of that substring (which is what we're doing above). How, specifically, does Ezoic modify the User-Agent? Does it always append X-Middleton to the end of the User-Agent? So, if it was present in the initial request, it would now appear twice?
    – MrWhite
    Mar 3 at 16:40
  • Or does Ezoic provide the original User-Agent string in a different HTTP response header? (A "reverse proxy" should not destroy elements of the original client request - so I suspect this information is still available somewhere.)
    – MrWhite
    Mar 3 at 17:09
  • The requests shows as follows 186.81.XXX.XXX - - [03/Mar/2021:19:55:27 +0000] "GET /es/xxxxx-xxxx-xxx HT TP/1.1" 200 11512 "https://example.com/es/xxxxx/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537 .36 X-Middleton/1" 13.66.XXX.XXX - - [03/Mar/2021:19:53:00 +0000] "GET /es/xxxxx-xxxx-xxxxx HTTP/1.1" 200 36675 "-" "Mozilla/5.0 (compatible; bingbot/2. 0; +http://www.bing.com/bingbot.htm) X-Middleton/1"
    – Irene
    Mar 3 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.