How can I check whether a Unicode codepoint is assigned or not?

user_16383578 0
Viewed 262

I am using Python 3 and I know all about hex, int, chr, ord, '\uxxxx' escape and '\U00xxxxxx' escape and Unicode has 1114111 codepoints...

How can I check if a Unicode codepoint is valid? That is, it is unambiguously mapped to a authoritatively defined character.

For example, codepoint 720 is valid; it is 0x2d0 in hex, and U+02D0 points to ː:

In [135]: hex(720)
Out[135]: '0x2d0'

In [136]: '\u02d0'
Out[136]: 'ː'

And 888 is not valid:

In [137]: hex(888)
Out[137]: '0x378'

In [138]: '\u0378'
Out[138]: '\u0378'

And 127744 is valid:

In [139]: chr(127744)
Out[139]: ''

And 0xe0000 is invalid:

In [140]: '\U000e0000'
Out[140]: '\U000e0000'

I have come up with a rather hacky solution: if a codepoint is valid, trying to convert it to a character will either result in the decoded character or the '\xhh' escape sequence, else it will return the undecoded escape sequence exactly same as original, I can check the return value of chr and check if it starts with '\u' or '\U'...

Now is the hacky part, chr doesn't decode invalid codepoints but it doesn't raise exceptions either, and the escape sequences will have length of 1 since they are treated as a single character, I have to repr the return value and check the results...

I have used this method to identify all invalid codepoints:

In [130]: invalid = []

In [131]: for i in range(1114112):
     ...:     if any(f'{chr(i)!r}'.startswith(j) for j in ("'\\U", "'\\u")):
     ...:         invalid.append(i)

In [132]: from pathlib import Path

In [133]: invalid = [(hex(i).removeprefix('0x'), i) for i in invalid]

In [134]: Path('D:/invalid_unicode.txt').write_text(',\n'.join(map(repr, invalid)))
Out[134]: 18574537

Can anyone offer a better solution?

unicode python-3.x python
3 Answers

I believe the most straight-forward approach is to use unicodedata.category(). The examples from the OP are unassigned codepoints, which have a category of Cn ("Other, not assigned").

>>> import unicodedata as ud
>>> ud.category('\u02d0')
'Lm'
>>> ud.category('\u0378')  # unassigned
'Cn'
>>> ud.category(chr(127744))
'So'
>>> ud.category('\U000e0000')  # unassigned
'Cn'

It also works for the control characters in the ASCII range:

>>> ud.category('\x00')
'Cc'

Further categories for invalid codepoints (according to comments) are Cs ("Other, surrogate") and Co ("Other, private use"):

>>> ud.category('\ud800')  # lower surrogate
'Cs'
>>> ud.category('\uf8ff')  # private use
'Co'

So a function for codepoint validity (as per the OP's definition) could look like this:

def is_valid(char):
    return ud.category(char) not in ('Cn', 'Cs', 'Co')

Important caveat: Python's unicodedata module embeds a certain version of Unicode, so the information is potentially out of date. For example, in my installation of Python 3.8, the Unicode version is 12.1.0, so it doesn't know about codepoints assigned in later versions of Unicode:

>>> ud.unidata_version
'12.1.0'
>>> ud.category('\U0001fae0')  # melting face emoji added in Unicode v14
'Cn'

If you need a more recent version of Unicode than the one of your Python version, you probably need to fetch an appropriate table directly from Unicode.

user_16984310
user_41770090

You may want to use the unicode-charnames library, independent from the Python core Unicode database. This package supports version 14.0 of the Unicode Standard (released September 14, 2021).

from unicode_charnames import charname, UNICODE_VERSION

assigned = []
control = []
private_use = []
surrogate = []
noncharacter = []
reserved = []
for x in range(0x110000):
    name = charname(chr(x))
    if not name.startswith("<"):  # code points assigned to an abstract character
        assigned.append(x)
    elif name.startswith("<control"):
        control.append(x)
    elif name.startswith("<private"):
        private_use.append(x)
    elif name.startswith("<surrogate"):
        surrogate.append(x)
    elif name.startswith("<noncharacter"):
        noncharacter.append(x)
    else:
        reserved.append(x)

print(f"# Unicode {UNICODE_VERSION}")
print(f"Code points assigned to an abstract character: {len(assigned):,}")
print(f"Unassigned code points: {len(reserved):,}")
print( "Code points with a normative function:")
print(f"   Control characters     -> {len(control):>7,}")
print(f"   Private-use characters -> {len(private_use):>7,}")
print(f"   Surrogate characters   -> {len(surrogate):>7,}")
print(f"   Noncharacters          -> {len(noncharacter):>7,}")

Output:

# Unicode 14.0.0
Code points assigned to an abstract character: 144,697
Unassigned code points: 829,768
Code points with a normative function:
   Control characters     ->      65
   Private-use characters -> 137,468
   Surrogate characters   ->   2,048
   Noncharacters          ->      66

The file you need is DerivedName.txt, available in the Unicode Character Database (UCD). The file for version 14.0 can be found here.

user_171744360
Related