TL;DR: When I try to set the index to a Pandas Nullable Integer type, it gets stored as the generic object type instead.
If I've got a dataframe setup like this:
import pandas as pd
df = pd.DataFrame({
'X': [1, 2, 3, 4, 5],
'A': [1, 2, None, 100, 200],
'B': [7, None, 77, None, 777],
},
dtype='Int64' # pandas Nullalbe Int type
)
df.dtypes
Out[1]:
X Int64
A Int64
B Int64
dtype: object
By default, the index is 0 to 4 and type int64:
In [2]: df.index.dtype, df.index.inferred_type, df.index.is_numeric()
Out[2]: (dtype('int64'), 'integer', True)
But when I set one of those Int64 columns ('X') to be the index, the index's dtype is object; even though it's inferred as numeric:
In [3]: df.set_index('X', drop=False, inplace=True)
...: df.index.dtype, df.index.inferred_type, df.index.is_numeric()
Out[3]: (dtype('O'), 'integer', True)
If I convert the column to a standard type (np.int) and set it as the index, then it works as per the normal/old(?) way:
In [4]: df['X'] = df['X'].astype('int64') # lowercase 'i'
...: df.set_index('X', drop=False, inplace=True)
...: df.index.dtype, df.index.inferred_type, df.index.is_numeric()
Out[4]: (dtype('int64'), 'integer', True)
I'm also not able to cast the index to a Nullable Int type:
In [5]: df.index.astype('Int64')
Out[5]: Index([1, 2, 3, 4, 5], dtype='object', name='X')
Works fine for Series
In [6]: pd.Series(df['X'].astype('Int64')).dtype
Out[6]: Int64Dtype()
...but not if I try to assign it to the index.
In [7]: df.index = pd.Series(df['X'].astype('Int64'))
...: df.index
Out[7]: Index([1, 2, 3, 4, 5], dtype='object', name='X')
If I try to the values then I get a deprecation warning; but still returns int64 (lowercase 'i' / numpy).
In [10]: df.index.values.astype('Int64')
<ipython-input-10-6e18e0684d5a>:1: DeprecationWarning: Numeric-style type codes are deprecated and will result in an error in the future.
df.index.values.astype('Int64')
Out[10]: array([1, 2, 3, 4, 5], dtype=int64)
Is this a bug/limitation of the new dtypes? If so, it's not specifically mentioned in the linked docs.