How to stop pandas from corrupting None, from optional float optional Int into floating with NAN?

Viewed 23

Creating a pandas.DataFrame corrupts column types, with Optional types in a way that changes the meaning of the values.

import pandas as pd
from dataclasses import dataclass
from typing import Optional


@dataclass
class ExampleData:
    a: int
    b: Optional[int] = None
    c: Optional[str] = None
    d: Optional[float] = None


x = ExampleData(3)
y = ExampleData(7, 6, 'yesterday', 4.5)
z = ExampleData(9, 400, 'zebra', 3.4)
df = pd.DataFrame([x, y, z])
df

Notice that the resulting int column "b" is cohersed into a float column, and all instances from "b" and "d" which should be Optional numerics have cast None into float('NaN'), which is fundamentally a different value.

    a   b       c           d
0   3   NaN     None        NaN
1   7   6.0     yesterday   4.5
2   9   400.0   zebra       3.4

What I want is to be able to preserve the int column and the None value entries

    a   b       c           d
0   3   None    None        None
1   7   6       yesterday   4.5
2   9   400     zebra       3.4

I notice that

df['b'].convert_dtypes(int)
0    <NA>
1       6
2     400
Name: b, dtype: Int64

shows how None could be preserved without corrupting the data types.

0 Answers
Related