Same word produces different Unicode normalization

Question

Problem:
I have a Hebrew word יֹשֵׁב that I want to convert to its Unicode value.

Using Python, I get the byte string:

\u05d9\u05b9\u05e9\u05b5\u05c1\u05d1

However, using this PHP website I get the following sequence:

%u05D9%u05B9%uFB2A%u05B5%u05D1

(if you copy from the word above from my post over to that website, you probably won't see this unicode and will see the same code that Python produces. I guess it is processed by stackoverflow before copying to that website).

Py code:

str = 'יֹשֵׁב'
print(str.encode('ascii', 'backslashreplace'))

I can see that the difference being שׁ is encoded differently. Python sees ש and the dot separately and the website sees שׁ as one unit. My question: Is there a way to get Python to produce the same result as that website does?

Steps taken:
I have read this post. I have tried the following codes, but they all produce the same result:

from unicodedata import normalize
S1 = 'יֹשֵׁב'
print(normalize('NFC', S1).encode('ascii', 'backslashreplace'))
print(normalize('NFD', S1).encode('ascii', 'backslashreplace'))
print(normalize('NFKC', S1).encode('ascii', 'backslashreplace'))
print(normalize('NFKD', S1).encode('ascii', 'backslashreplace'))

PS: I have no programming background. Learned Py myself. Pardon me if my question seems dumb.

Thank you very much!

Edit: the above message was typed in Safari. I've found out that Safari may twist the Unicode character value behind the Hebrew letter שׁ from '\ufb2a' to '\u05e9\u05c1'. But Chrome and Firefox (and possibly Edge) do not do the twist. So I'm now typing שׁ in Chrome to see if it twists or not.

user_235698 · Answer

Looking at UAX#15 of the Unicode standard, and that U+FA2B is on the composition exclusion list, it seems that character can't be generated by normalization. It can only be decomposed. The Python normalization is correct.

If you want that specific representation, you can use Unicode escapes:

s = '\u05D9\u05B9\uFB2A\u05B5\u05D1'

or by name:

s = '\N{HEBREW LETTER YOD}\N{HEBREW POINT HOLAM}\N{HEBREW LETTER SHIN WITH SHIN DOT}\N{HEBREW POINT TSERE}\N{HEBREW LETTER BET}'

user_4518341 · Answer

It seems like you're inputting the wrong version into Python. If it's in source code, it might be easier to use \uXXXX escapes instead of the Hebrew characters themselves. If not, I'm not sure where the problem would be; you'd need to provide a minimal reproducible example.

But both normalize to the same thing under all normalization forms.

Here's proof:

from unicodedata import normalize

forms = ['NFC', 'NFD', 'NFKC', 'NFKD']
norms = set()
for s in '\ufb2a\u05b5', '\u05e9\u05b5\u05c1':
    print(ascii(s))
    norm = {ascii(normalize(form, s)) for form in forms}
    #print(norm)  # If you want to show each set
    norms.update(norm)

print('Normals:', *norms)

Output:

'\ufb2a\u05b5'
'\u05e9\u05b5\u05c1'
Normals: '\u05e9\u05b5\u05c1'

Same word produces different Unicode normalization

2 Answers