How do I get str.translate to work with Unicode strings?

Daryl_Spitzer_4766 0
Viewed 34909

I have the following code:

import string
def translate_non_alphanumerics(to_translate, translate_to='_'):
    not_letters_or_digits = u'!"#%\'()*+,-./:;<=>?@[\]^_`{|}~'
    translate_table = string.maketrans(not_letters_or_digits,
                                       translate_to
                                         *len(not_letters_or_digits))
    return to_translate.translate(translate_table)

Which works great for non-unicode strings:

>>> translate_non_alphanumerics('<foo>!')
'_foo__'

But fails for unicode strings:

>>> translate_non_alphanumerics(u'<foo>!')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in translate_non_alphanumerics
TypeError: character mapping must return integer, None or unicode

I can't make any sense of the paragraph on "Unicode objects" in the Python 2.6.2 docs for the str.translate() method.

How do I make this work for Unicode strings?

unicode python string
7 Answers

I found that where in python 2.7, with type str, you would write

import string
table = string.maketrans("123", "abc")
print "135".translate(table)

whereas with type unicode you would say

table = {ord(s): unicode(d) for s, d in zip("123", "abc")}
print u"135".translate(table)

In python 3.6 you would write

table = {ord(s): d for s, d in zip("123", "abc")}
print("135".translate(table))

maybe this is helpful.

user_7149100

I had a unique problem compared to the others here. First I knew that my string possibly had unicode chars in it. (Thanks to Email on Mac...) But one of the common chars was the emdash AKA u"\u2014" character which needed to be converted (back) to two dashes AKA "--". The other chars that might be found are single char translations so they are similar to the other solutions.

First I created a dict for the emdash. For these I use a simple string.replace() to convert them. Other similar chars could be handled here too.

uTranslateDict = {
    u"\u2014": "--", # Emdash
}

Then I created a tuple for the 1:1 translations. These go through the string.translate() builtin.

uTranslateTuple = [(u"\u2010", "-"),  # Hyphen
                   (u"\u2013", "-"),  # Endash
                   (u"\u2018", "'"),  # Left single quote => single quote
                   (u"\u2019", "'"),  # Right single quote => single quote
                   (u"\u201a", "'"),  # Single Low-9 quote => single quote
                   (u"\u201b", "'"),  # Single High-Reversed-9 quote => single quote
                   (u"\u201c", '"'),  # Left double quote => double quote
                   (u"\u201d", '"'),  # Right double quote => double quote
                   (u"\u201e", '"'),  # Double Low-9 quote => double quote
                   (u"\u201f", '"'),  # Double High-Reversed-9 quote => double quote
                   (u"\u2022", "*"),  # Bullet
]

Then the function.

def uTranslate(uToTranslate):
    uTranslateTable = dict((ord(From), unicode(To)) for From, To in uTranslateTuple)
    for c in uTranslateDict.keys():
        uIntermediateStr = uToTranslate.decode("utf-8").replace(c, uTranslateDict[c])
    return uIntermediateStr.translate(uTranslateTable)

Since I know the format of the input string I didn't have to worry about two types of input strings.

user_42755940
Related