Pandas read from csv parsing incorrectly large integers

user_1682697 0
Viewed 1429

Hi i am having a problem. I am read from csv a file various columns and one of the columns is a 19 digit integer ID. The problem is if i just read it with no options the number is read as float. And in this case it seems to be mixing the numbers: For example the dataset has 100k of unique ID values but reading like that give me 10k unique values. I changed the read_csv options to read it as string and the problem remains while its being read as mathematical notation (eg: *e^18).

pd.set_option('display.float_format', lambda x: '%.0f' % x)
df=pd.read_csv(file)
pandas csv python
1 Answers

Asked, it really happens when you read BigInteger value from .scv via pd.read_csv. For example:

df = pd.read_csv('/home/user/data.csv', dtype=dict(col_a=str, col_b=np.int64))
# where both col_a and col_b contain same value: 107870610895524558

After reading following conditions are True:

df.col_a == '107870610895524558'
df.col_a.astype(int) == 107870610895524558
# BUT
df.col_b == 107870610895524560

Thus I suggest that in case of reading big integers it is possible to read them as string and then convert column type to int

user_129753460
Related