Change Pandas default NA type promotions

Viewed 161

I have this particular case where I have huge integers in input and the conversion in float64 actually change the value.

According to this page : https://pandas.pydata.org/pandas-docs/stable/user_guide/gotchas.html#na-type-promotions Pandas converts automatically integers with NA to float64.

The problem is when I have NA value in my inputs, pandas automatically cast it to 'float64' even if I specify 'Int64' in the constructor

I wrote a repl.it script with pandas 1.0.1 and python 3.8.1 showing my use case : https://repl.it/@RonanTREILLET/pandastestint

For those who don't want to play it, here is basically what I do :

test_df =[
    {'dataset_id': 327626200612520493}, 
    {'dataset_id': 327626200612520493}, 
]
df = DataFrame(data=test_df, columns=['dataset_id'])

then if I do

print(df[col_name])

the output is :

0    327626200612520493
1    327626200612520493
Name: dataset_id, dtype: int64

which is correct

but, if I do this :

print(df[col_name].astype('float').astype('Int64'))

I have this output :

0    327626200612520512
1    327626200612520512
Name: dataset_id, dtype: Int64

which is wrong.

The main problem is when you create dataframe with NA inputs like this :

#test with a NA value in DF, showing how Pandas automatically cast to `'float64'`
test_with_nan_df =[
    {'dataset_id': 327626200612520493}, 
    {'dataset_id': None}
]
df_w_nan = DataFrame(data=test_with_nan_df, columns=['dataset_id'])
print(df_w_nan[col_name].astype('Int64'))

or

df_w_nan_int64 = DataFrame(data=test_with_nan_df, columns=['dataset_id'], dtype='Int64')
print(df_w_nan_int64[col_name])

the output is the same :

0    327626200612520512
1                  <NA>
Name: dataset_id, dtype: Int64

As you may see, even if I force 'Int64' during the dataframe creation it's still the wrong value and I suppose pandas internally cast column to 'float' before casting it to 'Int64'

So IF my usage of Pandas is correct, I'd like to write a patch to override the NA type promotion's method.

Did someone already did this ? and is it even possible ?

Regards,

1 Answers

You can use object to ensure that pandas leaves the data unchanged. Then you can manually set the types.

pd.DataFrame(data=test_with_nan_df, columns=['dataset_id'], dtype='object').astype('Int64')
#           dataset_id
#0  327626200612520493
#1                 NaN

#dataset_id    Int64
Related