Limit search to latin characters

Question

I'd like to limit search to characters used on the English language + numbers. The reason is that looking at the slowest queries on mysql log i found most come from searches in Arab, Russian and Chinese characters, so i'd like to skip them and display an error message instead.

Answer 1

This solution filters search strings by applying a regular expression which only matches characters from the Common and Latin Unicode scripts.

---

Matching Latin Characters with Regular Expressions

I just had my mind blown over at Stack Overflow. As it turns out, regular expressions have a mechanism to match entire Unicode categories, including values to specify entire Unicode "scripts", each corresponding to groups of characters used in different writing systems.

This is done by using the \p meta-character followed by a Unicode category identifier in curly braces - so [\p{Common}\p{Latin}] matches a single character in either the Latin or Common scripts - this includes punctuation, numerals, and miscellaneous symbols.

As @Paul 'Sparrow Hawk' Biron points out, the u pattern modifier flag should be set at the end of the regular expression in order for PHP's PCRE functions to treat the subject string as UTF-8 Unicode encoded.

All together then, the pattern

/^[\p{Latin}\p{Common}]+$/u

will match an entire string composed of one or more characters in the Latin and Common Unicode scripts.

---

Filtering the Search String

A good place to intercept a search string is the pre_get_posts action as it fires immediately before WordPress executes the query. With more care, this could also be accomplished using a request filter.

function wpse261038_validate_search_characters( $query ) {
  // Leave admin, non-main query, and non-search queries alone
  if( is_admin() || !$query->is_main_query() || !$query->is_seach() )
    return;

  // Check if the search string contains only Latin/Common Unicode characters
  $match_result = preg_match( '/^[\p{Latin}\p{Common}]+$/u', $query->get( 's' ) );

  // If the search string only contains Latin/Common characters, let it continue
  if( 1 === $match_result )
    return;

  // If execution reaches this point, the search string contains non-Latin characters
  //TODO: Handle non-Latin search strings
  //TODO: Set up logic to display error message
}

add_action( 'pre_get_posts', 'wpse261038_validate_search_characters' );

---

Responding to Disallowed Searches

Once it's been determined that a search string contains non-Latin characters, you can use WP_Query::set() in order to modify the query by changing it's named query vars - thus affecting the SQL query WordPress subsequently composes and executes.

The most relevant query variables are probably the following:

• s is the query variable corresponding to a search string. Setting it to null or an empty string ('') will result in the WordPress no longer treating the query as a search - often times this results in an archive template displaying all posts or the front-page of the site, depending on the values of the other query vars. Setting it to a single space (' '), however, will result in WordPress recognizing it as a search, and thus attempting to display the search.php template.
• page_id could be used to direct the user to a specific page of your choice.
• post__in can restrict the query to a specific selection of posts. By setting it to an array with an impossible post ID, it can serve as a measure to ensure that the query returns absolutely nothing.

The above in mind, you might do the following in order to respond to a bad search by loading the search.php template with no results:

function wpse261038_validate_search_characters( $query ) {
  // Leave admin, non-main query, and non-search queries alone
  if( is_admin() || !$query->is_main_query() || !$query->is_seach() )
    return;

  // Check if the search string contains only Latin/Common Unicode characters
  $match_result = preg_match( '/^[\p{Latin}\p{Common}]+$/u', $query->get( 's' ) );

  // If the search string only contains Latin/Common characters, let it continue
  if( 1 === $match_result )
    return;

  $query->set( 's', ' ' ); // Replace the non-latin search with an empty one
  $query->set( 'post__in', array(0) ); // Make sure no post is ever returned

  //TODO: Set up logic to display error message
}

add_action( 'pre_get_posts', 'wpse261038_validate_search_characters' );

---

Displaying an Error

The way in which you actually display the error message is highly dependent on your application and the abilities of your theme - there are many ways which this can be done. If your theme calls get_search_form() in it's search template, the easiest solution is probably to use a pre_get_search_form action hook to output your error immediately above the search form:

function wpse261038_validate_search_characters( $query ) {
  // Leave admin, non-main query, and non-search queries alone
  if( is_admin() || !$query->is_main_query() || !$query->is_seach() )
    return;

  // Check if the search string contains only Latin/Common Unicode characters
  $match_result = preg_match( '/^[\p{Latin}\p{Common}]+$/u', $query->get( 's' ) );

  // If the search string only contains Latin/Common characters, let it continue
  if( 1 === $match_result )
    return;

  $query->set( 's', ' ' ); // Replace the non-latin search with an empty one
  $query->set( 'post__in', array(0) ); // Make sure no post is ever returned

  add_action( 'pre_get_search_form', 'wpse261038_display_search_error' );
}

add_action( 'pre_get_posts', 'wpse261038_validate_search_characters' );

function wpse261038_display_search_error() {
  echo '<div class="notice notice-error"><p>Your search could not be completed as it contains characters from non-Latin alphabets.<p></div>';
}

Some other possibilities for displaying an error message include:

• If your site uses JavaScript which can display "flash" or "modal" messages (or you add such abilities on your own), add to it the logic to display messages on page-load when a specific variable is set, then add a wp_enqueue_script hook with a $priority larger than that which enqueues that JavaScript, and use wp_localize_script() to set that variable to include your error message.
• Use wp_redirect() to send the user to the URL of your choice (this method requires an additional page load).
• Set a PHP variable or invoke a method which will inform your theme/plugin about the error such that it may display it where appropriate.
• Set the s query variable to '' instead of ' ' and use page_id in place of post__in in order to return a page of your choosing.
• Use a loop_start hook to inject a fake WP_Post object containing your error into the query results - this is most definitely an ugly hack and may not look right with your particular theme, but it has the potentially desirable side effect of suppressing the "No Results" message.
• Use a template_include filter hook to swap out the search template with a custom one in your theme or plugin which displays your error.

Without examining the theme in question, it's difficult to determine which route you should take.

Answer 2

You would do this by putting in a validation function in PHP to test the input against a regular expression like ^[a-zA-Z0-9,.!?' ]*

So it would look like this:

if ( preg_match( "^[a-zA-Z0-9,.!?'" ]*", {search variable} ) ) {
   // Success
} else {
   // Fail
}

The RexEx I used for all characters A-Z, a-z, 0-9, as well as ,, ., !, ?, ', ", and (space).

Answer 3

EDIT: This solution is not recommended

My solution below is a hack which abuses PHP's mbstring functions in an attempt to magically divine alphabets by looking at the arrangement of bytes which compose the string. This is a really bad idea and is highly prone to error.

Please see my other answer for a far simpler and much more reliable solution.

---

One means to prevent searches using non-Latin alphabets is to use PHP's mb_detect_encoding() function to see if the search string conforms to one of a custom selection of character encodings. A good place to do this is the pre_get_posts action, as it fires right before the query is executed.

What you actually do after you've determined a search is using an invalid encoding is really application specific. Here I've set the search query to a single space to ensure that WordPress still interprets the query as a search, and thus still loads the search.php template (and does not direct the user to the front-page, as happens when the search string is an empty string). I also take an added precaution of setting 'post__in' to an array with an impossible post ID in order to make sure that absolutely nothing is returned.

Alternately, you might consider setting the search string to null and setting page_id in order to direct the user to a page with your custom error message.

function wpse261038_validate_search_query_encoding( $query ) {
  $valid_encodings = array( 'Windows-1252' );

  // Ignore admin, non-main query, and non-search queries
  if( is_admin() || !$query->is_main_query() || !$query->is_seach() )
    return;

  // Retrieve the encoding of the search string (if it's one listed in $valid_encodings)
  $search_encoding = mb_detect_encoding( $query->get( 's' ), $valid_encodings, true );

  // If the search encoding is one in $valid_encodings, leave the query as-is
  if( in_array( $search_encoding, $valid_encodings ) )
    return;

  // If it wasn't, sabotage the search query
  $query->set( 's', ' ' );
  $query->set( 'post__in', array(0) );

  // Set up your error message logic here somehow, perhaps one of the following:
  // - Add a template_include filter to load a custom error template
  // - Add a wp_enqueue_scripts hook with a greater priority than your theme/plugin's, and
  //     use wp_localize_script() in the hook to pass an error message for your JavaScript
  //     to display
  // - Perform a wp_redirect() to send the user to the URL of your choice
  // - Set a variable with an error message which your theme or plugin can display
}

add_action( 'pre_get_posts', 'wpse261038_validate_search_query_encoding' );

---

Choosing Encodings

I wrote a coverage test comparing some dummy strings in different alphabets against all of the default encodings supported by PHP. It's not perfect by any stretch (I have no idea how realistic my dummy strings are, and it seems to choke on Japanese detection), but it's somewhat useful for determining candidates. You can see it in action here.

After researching potential character encodings flagged by that test, it seems like Windows-1252 is the perfect choice for your needs, covering the Latin alphabet as well as the accents for common Latin languages.

A selection of the ISO-8859 character sets should be another viable choice, however for reasons I can't wrap my head around, the mb_ functions don't seem to differentiate between ISO-8859's different character sets, despite listing them as separate encodings.

To allow some other common characters, you might also consider adding HTML-ENTITIES.

Answer 4

As I tried to explain to @MichaelRogers when he posted a similar question several days ago, knowing the character set (or script) used in a string is NOT sufficient to detect the language of that string.

Thus, while the method detailed by @bosco will remove Russian, etc strings (with the 2 corrections below), it will NOT limit your searches to English.

To see this, try:

$strings = array (
    'I\'m sorry',                   // English
    'Je suis désolé',               // French
    'Es tut mir Leid',              // German
    'Lorem ipsum dolor sit amet',   // Lorem ipsum
    'أنا سعيد',                     // Arabic
    'я счастлив',                   // Russian
    '我很高兴',                     // Chinese (Simplified)
    '我很高興',                     // Chinese (Traditional)
    ) ;
foreach ($strings as $s) {
    if (preg_match ('/^[\p{Latin}\p{Common}]+$/u', $s) === 1) {
        echo "$s: matches latin+common\n" ;
        }
    else {
        echo "$s: does not match latin+common\n" ;
        }
    }

[note: the 2 corrections mentioned above to what @bosco provided are:

1. the pattern is enclosed a string (required to be syntactically correct PHP)
2. added the /u modifier (required to treat pattern and subject as UTF-8 encoded, see PHP: Regex Pattern Modifiers]

which will produce:

I'm sorry: matches latin+common
Je suis désolé: matches latin+common
Es tut mir Leid: matches latin+common
Lorem ipsum dolor sit amet: matches latin+common
أنا سعيد: does not match latin+common
я счастлив: does not match latin+common
我很高兴: does not match latin+common
我很高興: does not match latin+common

[note: I speak English, French & some German (and a bit of Lorem ipsum :-), but relied on Google Translate for the Arabic, Russian and Chinese]

As you can see, relying on checking for the latin script will NOT ensure you have English.

There are a number of threads on StackOverflow (e.g., Detect language from string in PHP) that provide more info on the subject.