Why do Pandas integer `dtypes` not behave the same on Unix and Windows?

user_3585557 0
Viewed 452

In checking the dtypes for columns in a Pandas DataFrame, I realized that the data type for an integer column is np.int64, but what is surprising is that on Unix this equates to int but on Windows it is not. Why do they not behave the same? Is there a way to create the DataFrame in a way that the result is the same when compared using df.dtypes == int?

Here is some sample code to illustrate.

In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: pd.__version__
Out[3]: '1.0.1'

In [4]: np.__version__
Out[4]: '1.18.1'

In [5]: data = pd.DataFrame({'col_1': range(5), 'col_2': np.linspace(0, 1, 5)})

In [6]: data.dtypes
Out[6]: 
col_1      int64
col_2    float64
dtype: object

In [7]: data.dtypes == float
Out[7]: 
col_1    False
col_2     True
dtype: bool

All of that produces the same result on Windows and Unix, but if I compare the dtypes to int on Windows I get

In [8]: data.dtypes == int
Out[8]: 
col_1    False
col_2    False
dtype: bool

and on Unix I get

In [8]: data.dtypes == int
Out[8]:
col_1     True
col_2    False
dtype: bool

I tried specifying the datatypes. This works on Unix, I can input the datatypes by adding dtype=(int, float)

In [9]: data = pd.DataFrame({'col_1': range(5), 'col_2': np.linspace(0, 1, 5)}, dtype=(int, float))

In [10]: data.dtypes
Out[10]:
col_1      int64
col_2    float64
dtype: object

but on Windows this code fails with a ValueError

In [10]: data = pd.DataFrame({'col_1': range(5), 'col_2': np.linspace(0, 1, 5)}, dtype=(int, float))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-25-284f0f12d3b6> in <module>
----> 1 data = pd.DataFrame({'col_1': range(5), 'col_2': np.linspace(0, 1, 5)}, dtype=(int, float))

~\Miniconda3\envs\pandas_test\lib\site-packages\pandas\core\frame.py in __init__(self, data, index, columns, dtype, copy)
    423             data = {}
    424         if dtype is not None:
--> 425             dtype = self._validate_dtype(dtype)
    426
    427         if isinstance(data, DataFrame):

~\Miniconda3\envs\pandas_test\lib\site-packages\pandas\core\generic.py in _validate_dtype(self, dtype)
    257
    258         if dtype is not None:
--> 259             dtype = pandas_dtype(dtype)
    260
    261             # a compound dtype

~\Miniconda3\envs\pandas_test\lib\site-packages\pandas\core\dtypes\common.py in pandas_dtype(dtype)
   1872     # raise a consistent TypeError if failed
   1873     try:
-> 1874         npdtype = np.dtype(dtype)
   1875     except SyntaxError:
   1876         # np.dtype uses `eval` which can raise SyntaxError

ValueError: mismatch in size of old and new data-descriptor
pandas numpy python
3 Answers

pandas.DataFrame has a dtype parameter.

pandas.DataFrame(data=None, index=None, columns=None, dtype=None, copy=False)

You should be able to set a specific data type using that parameter.

user_145039550

For a platform agnostic means of comparing any type of integer or float, you can use

In [10]: [np.issubdtype(dtype, np.integer) for dtype in data.dtypes]
Out[10]: [True, False]

In [11]: [np.issubdtype(dtype, np.float) for dtype in data.dtypes]
Out[11]: [False, True]

The primary issue behind the Windows/Unix difference is that int equates to a pandas dtype of np.int64 on Unix and np.int32 on Windows.

This code illustrates the difference in behavior:

import numpy as np
import pandas as pd

print(f'numpy version: {np.__version__}')
print(f'pandas version: {pd.__version__}')

data = pd.DataFrame({
    'col_i': range(5),
    'col_f': np.linspace(0, 1, 5),
})
data['col_i32'] = data.col_i.astype(np.int32)
data['col_i64'] = data.col_i.astype(np.int64)
data['col_f32'] = data.col_i.astype(np.float32)
data['col_f64'] = data.col_i.astype(np.float64)
print(f'\ndata.dtypes: \n{data.dtypes}')
print(f'\ndata.dtypes == int: \n{data.dtypes == int}')
print(f'\ndata.dtypes == float: \n{data.dtypes == float}')

Here is the result on Windows:

numpy version: 1.18.1
pandas version: 1.0.1

data.dtypes:
col_i        int64
col_f      float64
col_i32      int32
col_i64      int64
col_f32    float32
col_f64    float64
dtype: object

data.dtypes == int:
col_i      False
col_f      False
col_i32     True  # the np.int32 column
col_i64    False
col_f32    False
col_f64    False
dtype: bool

data.dtypes == float:
col_i      False
col_f       True
col_i32    False
col_i64    False
col_f32    False
col_f64     True
dtype: bool

Here is the output on Unix:

numpy version: 1.18.1
pandas version: 1.0.1

data.dtypes:
col_i        int64
col_f      float64
col_i32      int32
col_i64      int64
col_f32    float32
col_f64    float64
dtype: object

data.dtypes == int:
col_i       True  # a np.int64 column
col_f      False
col_i32    False
col_i64     True  # a np.int64 column
col_f32    False
col_f64    False
dtype: bool

data.dtypes == float:
col_i      False
col_f       True
col_i32    False
col_i64    False
col_f32    False
col_f64     True
dtype: bool

Note that the reason specifying the dtype fails on Windows is because the types must have the same memory size. The pandas documentation states that "Only a single dtype is allowed." but this is clearly not the case because either of the following work on Windows and Unix.

data = pd.DataFrame({'col_i32': range(5), 'col_f32': np.linspace(0, 1, 5)}, dtype=(np.int32, np.float32))

data = pd.DataFrame({'col_i32': range(5), 'col_f32': np.linspace(0, 1, 5)}, dtype=(np.int64, np.float64))

I think what they actually meant is "Only a singe [data size] is allowed".

The issue for the error when specifying dtype=(int, float) goes back to the primary issues illustrated above, where Windows is casting int as np.int32 and float as np.float64, while Unix instead casts int as np.int64 and float as np.float64. Pandas requires they have the same memory size, which works out on Unix but not on Windows.

user_35855570 
>>> np.int64 == int
False
>>> np.int32 == int
False
>>> np.int == int
True
>>> np.int == np.int64
False
>>> np.int == np.int32
False

It is expected and I would stick with defining what the dtype you are using by specifying int64 or int32

user_54783730
Related