Float precision differs between elements in pandas dataframe

langness 0
Viewed 44

I am trying to read a dataframe from a csv, do some calculations with it and then export the results to another csv. While doing that I noticed that the value 8.1e-202 is getting changed to 8.1000000000000005e-202. But all the other numbers are represented correctly.

Example:

A example.csv looks like this:

id,e-value
ID1,1e-20
ID2,8.1e-202
ID3,9.24e-203

If I do:

import pandas as pd
df = pd.read_csv("example.csv")
df.iloc[1]["e-value"]
>>> 8.1000000000000005e-202

df.iloc[2]["e-value"]
>>> 9.24e-203

Why is 8.1e-202 being altered but 9.24e-203 isn't?

I tried to change the datatype that pandas is using from the default

df["e-value"].dtype
>>> dtype('float64')

to numpy datatypes like this:

import numpy as np
df = pd.read_csv("./temp/test", dtype={"e-value" : np.longdouble})

but this will just result in:

df.iloc[1]["e-value"]
>>> 8.100000000000000522e-202

Can someone explain to me why this is happening? I can't replicate this problem with any other number. Everything bigger or smaller than 8.1e-202 seems to work normally.

EDIT:
To specify my problem. I am aware that floats are not perfect. My actual problem with this is that once I write the dataframe back to a csv the resulting file will then look like this:

id,e-value
ID1,1e-20
ID2,8.1000000000000005e-202
ID3,9.24e-203

And I need the second row to be ID2,8.1e-202
I "fixed" this by just formatting this column before I write the csv, but I'm unhappy with this solution since the formatting will change other elements to something scientific notation where it was just a normal float.

def format_eval(e):
    return "{0:.1e}".format(e)

df["e-value"] = df["e-value"].apply(lambda x: format_eval(x))
dataframe pandas numpy python
1 Answers

Float number representation is something not so simple. Not every real number can be represented and almost all (relatively speaking) are actually approximations. Is not like integers, the precision varies and python has a precision undefined float really.

Each floating point standar will have their own set of real numbers that can represent exactly. There's no work around.

https://en.wikipedia.org/wiki/Single-precision_floating-point_format https://en.wikipedia.org/wiki/IEEE_754-2008_revision

If the problem really is the arithmetic or comparison, you should consider if error will grow or decrease. For example multiplying by large numbers can grow the representation error.

And also, when comparing you should do things like math.is_close. Basically comparing the distance between the numbers.

If you are trying to represent and operate real numbers, that aren't irrational numbers. Like integers, fractions or decimal numbers with finite digits, you can also consider cast to the proper digit representation like: int, decimal or fraction.

See this for further ideas: https://davidamos.dev/the-right-way-to-compare-floats-in-python/#:~:text=How%20To%20Compare%20Floats%20in%20Python&text=If%20abs(a%20%2D%20b),rel_tol%20keyword%20argument%20of%20math.

Franco Milanese0
Related