How does the jupyter notebook parse unicode variable names and why do I get a bug?

user_5998704 0
Viewed 796

One thing that I quite like working in Python inside jupyter notebook is that I can use some unicode symbols to name my variables. For example, to use greek letters, I type \alpha followed by tab and I get α.

I just ran into an unexpected behaviour when using a bold capital T, \bfT followed by tab which results in .

The experiment is the following. Inside a cell (running Python 3) type: 

T = 1
 = 2
print(T)  # prints 2

To my surprise, the second line is reassigning the variable T but I would expect it to be different from . Can somebody please explain what's the catch with using Unicode?

I don't know if it helps, but as another experiment, I can see that the same two symbols as strings are in fact different

'T'.encode('utf8'), ''.encode('utf8')  # (b'T', b'\xf0\x9d\x90\x93')

How is the notebook processing my variable names?

jupyter-notebook unicode python
1 Answers

This behaviour is defined in the python language specification for identifiers (variable names). https://docs.python.org/3/reference/lexical_analysis.html#identifiers

2.3. Identifiers and keywords

Identifiers (also referred to as names) are described by the following lexical definitions.

The syntax of identifiers in Python is based on the Unicode standard annex UAX-31, with elaboration and changes as defined below; see also PEP 3131 for further details.

[...]

All identifiers are converted into the normal form NFKC while parsing; comparison of identifiers is based on NFKC.

We can confirm that T and are equivalent under NFKC using the standard library unicodedata module.

>>> import unicodedata
>>> unicodedata.normalize('NFKC','') == 'T'
True

So you should avoid using so similar unicode characters in the same scope as unique variable names.

But there's still a lot of unicode characters that are unique and can be safely used in identifiers:

>>> unicodedata.normalize('NFKC','λ')
'λ'
user_19778470
Related