Equality of Equivalent Unicode Strings

Prem_3057742 0
Viewed 69

I have a question about Equality of Equivalent Unicode Strings, and I have given some Python code , but it may be a general question for all programming languages like C , C++ , Perl or Java.

I have a string u, which displays a glyph, which is 05D1 (HEBREW LETTER BET) incorporating 05BC (HEBREW POINT DAGESH) and 05B8 (HEBREW POINT QAMATS)

I have another string v, which also displays the exact same glyph, which is 05D1 (HEBREW LETTER BET) incorporating 05B8 (HEBREW POINT QAMATS) and 05BC (HEBREW POINT DAGESH)

Difference is in the order of incorporating or combining the "POINTS". Many other languages and letters in Unicode can incorporate or combine with other POINTS to generate a glyph.

Now, u and v are practically and visually the same glyph, but can not be compared naively in Python:

>>> u='\u05D1\u05BC\u05B8'
>>> v='\u05D1\u05B8\u05BC'
>>> u
'בָּ'
>>> v
'בָּ'
>>> u==u
True
>>> v==v
True
>>>
>>> u==v
False
>>> v==u
False
>>> u is v
False
>>>

When I make a webpage with these glyphs, view the webpage and search for a glyph , even Mozilla FireFox treats these two as Distinct, but Google Chrome treats them as same. Obviously, Python and FireFox check for byte-equality, while Chrome is not checking byte-equality, but somehow figuring out that both are same.

What is the algorithm to check Equality of Equivalent Unicode Strings in Python ?

unicode character-encoding algorithm python string
1 Answers

The Python library unicodedata can solve such problems.

You would be using code similar to:

import unicodedata as ud

astr=u"\N{LATIN SMALL LETTER E}" + u"\N{COMBINING ACUTE ACCENT}"
combined_astr=ud.normalize('NFC',astr)

Normalizing to NFC puts certain sequences of combining marks into a well-defined order, which will probably solve the problem of comparing such composed characters. Other normalization options will discard combining characters so that you may compare the raw characters.

For more information and examples see:

harrymc_1653580
Related