How to get all characters of an arbitrary encoding?

user_962190 0
Viewed 521

If I want to know which letters are part of the ascii charset, I can simply ask python, which is nice:

>>> import string
>>> string.ascii_letters
'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'

I searched for a while, but couldn't find a generic function that returns charsets of arbitrary encodings. Something like this:

>>> import string
>>> string.get_charset('latin1')  # doesn't exist =(
'abc ... äöü ...'

Or did I just miss it? A function that checks whether a string only contains characters of some encoding would work too, but I'd like the intuitiveness of just having all valid characters as a list.

unicode character-encoding python-3.x python
1 Answers

As far as I'm aware, no such function exists in the standard library.

In lack of a better idea, here's an ugly hack that tries to encode every single character in the utf8 range with the specified encoding and removes those characters that couldn't be encoded:

def get_charset(encoding):
    all_chars = ''.join(chr(x) for x in range(0x110000))
    return all_chars.encode(encoding, errors='ignore').decode(encoding)

Output:

>>> get_charset('latin-1')
'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0¡¢£¤¥¦§¨©ª«¬\xad®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ'

Speed test:

In [2]: %timeit get_charset('latin1')
306 ms ± 8.34 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
user_12229510
Related