Strange characters - despite everything being UTF-8

Question

Not sure why this thread was closed, but this is the same issue inflicting many people.

All my WP config settings are in order:

//define('DB_CHARSET', 'utf8');
//define('DB_CHARSET', 'utf8_unicode_ci');
//define('DB_COLLATE', '');

I even tried enabling them one by one. None worked.

When I save a post, weird characters appear in place of apostrophes and spaces. This happens whether I type content manually or

I've tried a few plugins.

1. UTF-8 Sanitize
2. Convert WP to UTF-8

..etc.

None of them work. The problem persists.

I've also changed the database's character set and collation in MYSQL. Screenshot:

This is a screenshot of me entering some content by pasting it:

But immediately upon saving, the text comes back with garbled characters having replaced it:

What else?

I went through the rigmarole of dumping the entire MySQL DB, then removing all older non-utf8 characters via TR command.

tr -cd '\11\12\15\40-\176' < file-with-binary-chars > clean-file

Where the file-with-binary-chars was the MySQL dump. Then I restored the table.

My MySQL config is all utf8:

[client]
default-character-set=utf8

[mysqld]
character-set-client=utf8
collation-server=utf8_unicode_ci
character-set-server=utf8

My browser is Chrome. The encoding is UTF-8 (in the VIEW menu).

What else can I do? Do I need to make all the plugin files utf-8 as well?

FYI, this blog is one of the Wordpress blogs. There are other newer Wordpress installations on the same server using the same installation of MySQL 5.6.17, but they don't have such an issue. My guess is that this being an older blog may have some variations in the text entered long ago, but frankly after having done all of the above, I really don't know what else I can do.

Thanks for any inputs or pointers!

Answer 1

This is typically caused when you are copying/pasting MS Word information into the WordPress content editor. WordPress uses something called "Smart Quotes", via a function named wptexturize().

Ideal Solution

The ideal solution would be to go back through your content, and replace all single/double quotes using the keyboard.

However, if you're working with massive copy/pastes, this may not be feasible.

Disable wptexturize() Filter

Another option is to disable the wptexturize() filter from running; which can be accomplished by placing the following code in your child theme functions.php file:

remove_filter('the_content', 'wptexturize');

You may also wish to remove the filter from comments and/or excerpts:

remove_filter('comment_text', 'wptexturize');
remove_filter('the_excerpt', 'wptexturize');

Or for titles:

remove_filter ('single_post_title', 'wptexturize');
remove_filter ('the_title', 'wptexturize');
remove_filter ('wp_title', 'wptexturize');

Clean Database

For existing content which has already saved the "weird" characters into the database; you may need to clean the database by running the following queries from PHPMyAdmin (be sure to take a database backup first):

UPDATE wp_posts SET post_content = REPLACE(post_content, '“', '“');
UPDATE wp_posts SET post_content = REPLACE(post_content, 'â€', '”');
UPDATE wp_posts SET post_content = REPLACE(post_content, '’', '’');
UPDATE wp_posts SET post_content = REPLACE(post_content, '‘', '‘');
UPDATE wp_posts SET post_content = REPLACE(post_content, '—', '–');
UPDATE wp_posts SET post_content = REPLACE(post_content, '–', '—');
UPDATE wp_posts SET post_content = REPLACE(post_content, '•', '-');
UPDATE wp_posts SET post_content = REPLACE(post_content, '…', '…');

Plugins

Well... it's WordPress. You can always use a plugin to help manage the wptexturize() filter. Take a look through This List, and see if one is right for you.

Answer 2

I was facing the same exact problem.

I tried everything just like you and at the end I tried one last thing.

Changed the DB_CHARSET in wp-config file to latin1.

And things worked :/

define('DB_CHARSET', 'latin1');
define('DB_COLLATE', 'utf8mb4_general_ci');

Answer 3

In another case, if you are using PHP Dom (loadHTML) somewhere, there is a need to load HTML as UTF-8. I have fixed it by:

Replacing

@$dom->loadHTML($html);

to

@$dom->loadHTML('<?xml encoding="UTF-8">' . $html);

Answer 4

The following fixed it for me, so posting here to share. Even though I ran the SQL updates suggested, I was still getting the funny symbols before the "read more" links.

I'm totally shocked. After hours of trying things and testing, I finally found the "Settings" then "Readings" option below: "Encoding for pages and feeds". After changing from UTF-7 to UTF-8, everything looks good again.

And even stranger, after changing it to UTF-8, the option disappears from the page. According to WordPress site, this option was removed since release 3.5.

Answer 5

The issue can cause by the plugins and direct FTP uploads.

Here's the detail:

http://iiiji.com/wordpress-appearing-weird-characterscode/

Answer 6

In my case the strange characters weren't written to the database. Check if the raw SQL is showing the incorrect characters as well. If you're seeing regular apostrophes in the database, comment out the following two lines in the wp-config.php:

// define('DB_CHARSET', 'utf8');
// define('DB_COLLATE', '');

That fixed it for me right away.

"This character encoding problem can happen after a database upgrade" - Link