I have a set of unicode codepoints stored as integers, and I'd like to encode these as UTF-8. If I understand correctly, UTF-8 is just an encoding for integers (the fact that it's used for unicode in particular isn't fundamental to UTF-8), so this should just be a matter of encoding an integer in the UTF-8 encoding. Is there a standard utility for doing this, and if not, is there an easy way of doing it manually?
2 Answers
There is an easy way. If you are on windows you might run into problems if you surpass UTF-16LE characters due to platform limitations. On linux you should be safe with full unicode.
>>> my_unicode_codepoints = [1234, 2345, 3456, 4576] # example codepoints
>>> [chr(i) for i in unicode_codepoints] # step 1: use python automagic for casting to wide enough chars
['Ӓ', 'ऩ', '\u0d80', 'ᇠ']
>>> "".join([chr(i) for i in unicode_codepoints]) # step 2: join to string
'Ӓऩ\u0d80ᇠ'
>>> "".join([chr(i) for i in unicode_codepoints]).encode("utf-8") # step 3: encode your string
b'\xd3\x92\xe0\xa4\xa9\xe0\xb6\x80\xe1\x87\xa0'
The last line is the result you are looking for.
this might be self explanatory
[ord(c) for c in ('a', 'ö', '')]
>>> [97, 246, 129303]
[chr(n) for n in [97, 246, 129303]]
>>>> ['a', 'ö', '']
Related