Creating a pandas.DataFrame corrupts column types, with Optional types in a way that changes the meaning of the values.
import pandas as pd
from dataclasses import dataclass
from typing import Optional
@dataclass
class ExampleData:
a: int
b: Optional[int] = None
c: Optional[str] = None
d: Optional[float] = None
x = ExampleData(3)
y = ExampleData(7, 6, 'yesterday', 4.5)
z = ExampleData(9, 400, 'zebra', 3.4)
df = pd.DataFrame([x, y, z])
df
Notice that the resulting int column "b" is cohersed into a float column, and all instances from "b" and "d" which should be Optional numerics have cast None into float('NaN'), which is fundamentally a different value.
a b c d
0 3 NaN None NaN
1 7 6.0 yesterday 4.5
2 9 400.0 zebra 3.4
What I want is to be able to preserve the int column and the None value entries
a b c d
0 3 None None None
1 7 6 yesterday 4.5
2 9 400 zebra 3.4
I notice that
df['b'].convert_dtypes(int)
0 <NA>
1 6
2 400
Name: b, dtype: Int64
shows how None could be preserved without corrupting the data types.