WordPress and TinyMCE both recently have been updated to allow you to modify the special characters (charmap) without having to hack the core. You can lookup the symbols you want from https://dev.w3.org/html5/html-author/charref and then either add the following code to your theme's functions.php or make a plugin:
add_filter( 'tiny_mce_before_init', 'tinymce_add_chars' );
function tinymce_add_chars( $settings ) {
$new_chars = json_encode( array(
array( '8224', 'Dagger' ),
array( '8230', 'Horizontal ellipsis' ),
array( '8539', '1/8 Fraction' ),
array( '8730', 'Square Root' ),
array( '8818', 'Less than or equivalent to' ),
array( '8819', 'Greater than or equivalent to' ),
array( '0963', 'Sigma' ),
array( '0956', 'Mu' ),
) );
$settings['charmap_append'] = $new_chars;
return $settings;
}
This modification relies on the tiny_mce_before_init hook in WordPress and TinyMCE's charmap_append settings. When adding elements you add each symbol as an array where the first item is the HTML entity code and the second is the description.
You can see in the screenshot below that my symbols have been appended to the existing list of symbols.
It looks like it's possible to also override the default list of symbols by using TinyMCE's charmap
setting and listing the symbols you want.