s1='ফটিকছড়ি' #escape-unicode= %u09AB%u099F%u09BF%u0995%u099B%u09A1%u09BC%u09BF
s2='ফটিকছড়ি' #escape-unicode= %u09AB%u099F%u09BF%u0995%u099B%u09DC%u09BF
They are looking the same but are different. How can I consider them as the same string?
How to compare Bengali homoglyph words or characters in Python?
Viewed 39
1 Answers
In Unicode, the character U+09DC is canonically equivalent to the sequence U+09A1 U+09BC. When you compare Unicode strings, you should always use Unicode normalization to fold together canonically equivalent sequences. So, convert both strings to Unicode normalization form C or Unicode normalization form D before comparing.
See UAX #15 Unicode Normalization Forms for details on Unicode normalization.
See this answer for how to normalize Unicode strings in Python.
Related