Convert angle bracketed embedded UTF-8 hex values to accented character

user_1422451 0
Viewed 211

In pandas read_json, UTF-8 accented or diatric characters are converted to angle brackets of corresponding hex values. How can this be avoided or fixed to render actual UTF-8 character value?

Consider the following example that pulls from an S3 bucket of all current R GitHub CRAN packages. Notice output of accented characters are represented by angle bracketed hex values:

import pandas as pd

# S3 (REQUIRES s3fs)
json_df = pd.read_json("s3://public-r-data/ghcran.json")

# URL (NO REQUIREMENT)
json_df = pd.read_json("http://public-r-data.s3-website-us-east-1.amazonaws.com/ghcran.json")

json_df.loc[884, "Title"]
# Misc Functions of Eduard Sz<c3><b6>cs

json_df.loc[213, "Author"]
# Kirill M<c3><bc>ller [aut, cre]

json_df.loc[336, "Maintainer"]
# H<c3><a9>l<c3><a8>ne Morlon <morlon@biologie.ens.fr>

To avoid a mapping dictionary replace solution for these hex values, is there a compact solution to avoid or fix such embedded hex values to actual UTF-8 characters? Specifically, how can above results be converted to the following:

# Misc Functions of Eduard Szöcs

# Kirill Müller [aut, cre]

# Hélène Morlon <morlon@biologie.ens.fr>
pandas hex encoding python string
1 Answers

Ah... the pain of incorrectly encoded strings! I once encountered a similar problem and here's an adapted version of how I solved it.

It breaks the original string into a byte array, then combine some bytes together, and re-encode the new array as UTF-8. It is far from foolproof but may not be a problem depends on your requirements:

def reencode(string):
    def is_hex(i):
        return (48 <= i <= 57) or (97 <= i <= 102)
    
    def to_bytes(arr):
        if len(arr) != 4:
            return None
        
        a, b, c, d = arr
        if a == 60 and is_hex(b) and is_hex(c) and d == 62:
            return bytes.fromhex(chr(b) + chr(c))

        return None

    old = string.encode('ascii')
    new = bytearray()

    i = 0
    while i < len(old):
        b = to_bytes(old[i:i+4])
        if b:
            new.extend(b)
            i += 4
        else:
            new.append(old[i])
            i += 1
    return new.decode('utf8')

Test:

df = pd.DataFrame([
    'Misc Functions of Eduard Sz<c3><b6>cs',
    'Kirill M<c3><bc>ller [aut, cre]',
    'H<c3><a9>l<c3><a8>ne Morlon <morlon@biologie.ens.fr>'
], columns=['name'])
df['name'].apply(reencode)

Output:

0            Misc Functions of Eduard Szöcs
1                  Kirill Müller [aut, cre]
2    Hélène Morlon <morlon@biologie.ens.fr>
user_25389390
Related