I have this particular case where I have huge integers in input and the conversion in float64 actually change the value.
According to this page : https://pandas.pydata.org/pandas-docs/stable/user_guide/gotchas.html#na-type-promotions Pandas converts automatically integers with NA to float64.
The problem is when I have NA value in my inputs, pandas automatically cast it to 'float64' even if I specify 'Int64' in the constructor
I wrote a repl.it script with pandas 1.0.1 and python 3.8.1 showing my use case : https://repl.it/@RonanTREILLET/pandastestint
For those who don't want to play it, here is basically what I do :
test_df =[
{'dataset_id': 327626200612520493},
{'dataset_id': 327626200612520493},
]
df = DataFrame(data=test_df, columns=['dataset_id'])
then if I do
print(df[col_name])
the output is :
0 327626200612520493
1 327626200612520493
Name: dataset_id, dtype: int64
which is correct
but, if I do this :
print(df[col_name].astype('float').astype('Int64'))
I have this output :
0 327626200612520512
1 327626200612520512
Name: dataset_id, dtype: Int64
which is wrong.
The main problem is when you create dataframe with NA inputs like this :
#test with a NA value in DF, showing how Pandas automatically cast to `'float64'`
test_with_nan_df =[
{'dataset_id': 327626200612520493},
{'dataset_id': None}
]
df_w_nan = DataFrame(data=test_with_nan_df, columns=['dataset_id'])
print(df_w_nan[col_name].astype('Int64'))
or
df_w_nan_int64 = DataFrame(data=test_with_nan_df, columns=['dataset_id'], dtype='Int64')
print(df_w_nan_int64[col_name])
the output is the same :
0 327626200612520512
1 <NA>
Name: dataset_id, dtype: Int64
As you may see, even if I force 'Int64' during the dataframe creation it's still the wrong value and I suppose pandas internally cast column to 'float' before casting it to 'Int64'
So IF my usage of Pandas is correct, I'd like to write a patch to override the NA type promotion's method.
Did someone already did this ? and is it even possible ?
Regards,