Pandas read_csv with integer columns with null values without precision loss

user_9393952 0
Viewed 408

I have a csv column with integers that has nulls in it, that I want to read with pandas. In the sample file below the column nr_ok has the same content that the column nr_nan, except for the missing value in the first row

row,nr_ok,nr_nan
1,4696374908103381,
2,1780963748798374342,1780963748798374342
3,719826117241460269,719826117241460269

I'm working with pandas version 1.3.4. If I attempt to read it without dtype specification, pandas converts the second column to float and I lose precision. So I tried specifiying dtype, using the Nullable type 'Int64'

type_dict = {'nr_nan':'Int64'}
df2 = pd.read_csv('test_int64.csv', dtype = type_dict)
df2

As a result I still lose precision. Notice how the last digits of the numbers have changed compared to the original

  | row |               nr_ok |              nr_nan
------------------------------------------------------  
0 |   1 |    4696374908103381 |                <NA>
1 |   2 | 1780963748798374342 | 1780963748798374400
2 |   3 |  719826117241460269 |  719826117241460224

It seems to me that pandas is reading the column as float and converting it to integer after that, with the subsequent precision loss.
Is there any way I can read a file with integer columns containing null values without losing precision?

pandas nullable integer python
1 Answers

Try using:

df2 = pd.read_csv('test_int64.csv', converters={'nr_nan':lambda x: pd.NA if x == '' else int(x)})
df2.nr_nan = df2.nr_nan.astype('Int64')

The first line will use a custom converter and will result in a mixed type field (a mix of 64 bit integers and pd.NA) The second line converts to the nullable int type

user_32301330
Related