If you create a post whose title has an accent in it and some other unicode character in it, eg à 漢語 title thing, its slug (permalink) will become a-漢語-title-thing... ie, the à was converted into a regular a, but those unicode Chinese characters were left intact. Why doesn't WordPress leave the accented characters alone? I created a code snippet to tell WordPress to leave them alone

function mn_sanitize_title($modified_title, $original_title, $context)
{
    // the $modified_title may have had accents removed, but not the $original_title
    return $original_title;
}
// set this filter to run BEFORE WP already ran the title through `sanitize_title_with_dashes`
add_filter('sanitize_title', 'mn_sanitize_title', 5, 3);

and it seems to work fine (accented characters are left intact in post slugs) so I'm wondering why WordPress developers removed accented characters from post slugs in the first place?

  • I would note that you may have removed some security features in the process, <script> may now be a valid post slug. I would also note that none of the non-ascii characters are valid as URLs without specialised encoding – Tom J Nowell Aug 8 at 23:24
  • Thanks for chiming in Tom! But one really important note: my code snippet runs BEFORE WP runs the slug through sanitize_title_with_dashes. So it's not totally circumventing all of the post slug sanitization- just the part that removes the accented characters. The slug still gets run through all of sanitize_title_with_dashes. So it removes scripts, and encodes the non-ascii characters. I've tested it and found that's what it does... So I suppose I should clarify my question- why are accented characters not just percent-encoded like all other non-ascii characters? – thespacecamel Aug 9 at 3:13
  • 1
    "why something is like that" is a very poor type of question. Unless there is a good documentation (in which case you would have probably found it yourself) or you know the person that wrote the code, it is basically impossible to do more than guessing. – Mark Kaplun Aug 9 at 4:24
  • Thanks @MarkKaplun for the feedback. Where would you suggest discussing this? Or how should I have phrased the question? I suppose I was hoping someone out there knew the person who wrote the code, or was more familiar with the history of WordPress' code, would have some helpful info. But yes, having an exact answer, like "because otherwise, X would have happened" might be difficult. – thespacecamel Aug 10 at 4:27
  • you can ask on the wordpress slack, but I will be very surprised if anyone will know without digging in the history of trac tickets and maybe even then... – Mark Kaplun Aug 10 at 6:18
up vote 3 down vote accepted

The sanitize_title() function uses remove_accents() right away. Both of these functions date way back to <= v1.2.1. The remove_accents() function is a hard coded list of accent characters to explicitly replace a handful of characters in a few specific languages. The inline comment and function reference simply say:

Converts all accent characters to ASCII characters.

According to RFC 3986, valid URLs are only ASCII.

So (although I can't find any evidence of this) I assume what's going on here is that the accent characters are replaced to make a ascii-valid URL for languages that have just a few almost-ascii characters. And that this originates from way back.

Invalid URL /à-b-c/ becomes valid /a-b-c/ (instead of valid encoded /%wtv-b-c/ title).


As to why the Chinese characters (which aren't ASCII) are not replaced, striped out, or encoded/escaped by WordPress seems to be intentional. Again, I can't find any documentation on this, but the characters aren't even close to ascii-valid like aforementioned accent characters are, so there's nothing to replace them with. Escaping the URL would be ridiculous, nearly unusable. And this entire thread, and notably this post, shed some light on these characters in URLs:

[addresses with non-ASCII characters] are not URIs (and therefore not URLs, since URLs are a type of URIs). If we consider ourselves beholden to the terminology of existing IETF standards, then we should properly call them IRIs (Internationalized Resource Identifiers), as defined in RFC 3987, which are technically not URIs but can be converted to URIs simply by percent-encoding all non-ASCII characters in the IRI.

So I would assume with WordPress not having handlers for these characters, it's leaving it up to the user and the browser in these cases, but it's been cleaning up accents since the early days.

(I realize this answer doesn't satisfy the question, but hopefully it provides a bit more info to get you closer to your answer).

  • 1
    not to dispute the answer, but I want to point out that 1.2 was ages ago and the modern day browsers might support non ascii characters much better than they have done back then, and therefor whatever was the reason might not apply any more. – Mark Kaplun Aug 9 at 4:21
  • maybe I should be more specific. While URLs are ASCII, everything which is not can be decoded into ASCII and will be displayed "as intended" in the browser. – Mark Kaplun Aug 9 at 4:26
  • @MarkKaplun the RFC dictates that non-ascii characters aren't supported, hence URL encoding. We get this every few months when a developer is working with Arabic or Kanji and sees encoded values in URLs and gets confused why WP isn't outputting proper characters as entered in slugs – Tom J Nowell Aug 9 at 12:42
  • @TomJNowell that is not a "reason". Maybe a reason is that browsers do not support the full unicode character set properly in the address bar, but one may claim that this should be left to the user to decide if he is willing to live with it as it might have impact on software that does better content parsing like google (not arguing that filtering is wrong, just saying that it is hard to know the original reason, for example at that time hebrew support was horrible and my clients preferred to avoid using "pretty urls") . – Mark Kaplun Aug 9 at 14:05
  • 1
    They do, hence how encoded values are displayed as arabic or chinese characters in the address bar. URLs simply don't do unicode characters because that's the standard that was agreed upon. If you send a HTTP request with unicode characters it will get punted or converted to fit the RFC. There's literally an ISO standard that forbids it, and another ISO standard on how to fit non-ASCII characters in URLs – Tom J Nowell Aug 9 at 14:51

Since we already have one "answer" I will add an "answer" of my own, although it is no more than a guess as well.

The important aspect is the state of i18n 10+ years ago, and this applies not only to the web related software but also to the OSs themselves. Back then IDN was not yet a standard and the web was ASCII oriented, and it would not have done the web much good to be better at i18n because almost no major OS supported it out of the box, and if for example you wanted windows that support hebrew you had to buy a special version, or install a language pack. But even if you had all the appropriate language packs installed, browers had bug and in the era before chrome it would have taken years before browsers had new versions, and web servers also had bugs in properly processing URL and properly pass them to the PHP layer.

The solution for latin scripts was easy, just convert anything which is not ascii to phonetically correct ASCII equivalence. It is very telling that the code do not handle hebrew although it has similar accent problems. My guess about hebrew is that both accents are less important in the normally read text (kids learning at young age to "imagine" accents based on understand the word in its context, and the places where you actually need them is when a text in foreign language is using the hebrew script for example yiddish), and that by the time the web got really going in israel those bugs were mostly fixed (although there was some period of time in which people preferred to avoid the problem by not using pretty permalinks which became default only in 4.4, or use slugs in english).

Now day I do not think that function serves anything and like other parts of wordpress code it is maintained as part of inertia and not based on actual need.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.