UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-15: surrogates not allowed

user_11621273 0
Viewed 10647

I am having trouble trying to print the result of a unicode here's what i have tried

data = u"\ud835\udc6a\ud835\udc89\ud835\udc90\ud835\udc84\ud835\udc8c"
result = data.encode('utf-8', 'surrogatepass') 
#b'\xed\xa0\xb5\xed\xb1\xaa\xed\xa0\xb5\xed\xb2\x89\xed\xa0\xb5\xed\xb2\x90\xed\xa0\xb5\xed\xb2\x84\xed\xa0\xb5\xed\xb2\x8c'
result.decode('utf-8')
#UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte

According to Charbase, \udc90 is an invalid character https://charbase.com/dc90-unicode-invalid-character

I get to convert the unicode through this website: https://www.online-toolz.com/tools/text-unicode-entities-convertor.php under the section "Decode/Unescape Unicode Entities"

Here's a screenshot of the result

enter image description here

How can i print this unicode out ? I am receiving data from an API and wants to store it in MySQL database. Currently, the result in MySQL database is ???????????

utf-8 encoding unicode python-3.x python
1 Answers

What that website presents are likely JSON format escape sequences containing surrogate pairs, which are really a UTF-16 thing, which Javascript treats strings as under the hood. The same raw string literals aren't valid in Python. What you want is not to let Python interpret the escape sequences, but create a string which contains escape sequences:

>>> r'\ud835\udc6a\ud835\udc89\ud835\udc90\ud835\udc84\ud835\udc8c'
'\\ud835\\udc6a\\ud835\\udc89\\ud835\\udc90\\ud835\\udc84\\ud835\\udc8c'

Since this is Javascript/JSON format, use the json module to decode it:

>>> import json
>>> json.loads(r'"\ud835\udc6a\ud835\udc89\ud835\udc90\ud835\udc84\ud835\udc8c"')
''

The way Python would encode this string as escape sequences is:

>>> print(''.encode('unicode-escape').decode('ascii'))
\U0001d46a\U0001d489\U0001d490\U0001d484\U0001d48c
user_4760
Related