I want to create a pandas dataframe from a json string. To be able to flatten the nested json source data I need to use pandas.json_normalize() (instead of pandas.read_json()). In the source data some of the columns are integer type and sporadically have missing values. This causes the column to be "upcast" to float:
import pandas as pd
data = [
{ "c1": 1, "c2": 1, "c3": 1.01 },
{ "c1": 2, "c3": 2.02 },
{ "c1": 3, "c2": 3, "c3": 3.03 }
]
df = pd.json_normalize(data)
print(df, "\n\n", df.dtypes)
which results in
c1 c2 c3
0 1 1.0 1.01
1 2 NaN 2.02
2 3 3.0 3.03
c1 int64
c2 float64
c3 float64
(I have stripped down the example to the crux of the matter. In reality the source data is far more complex.)
Applying df.convert_dtypes() kind of solves the problem after the fact but the function feels a bit creepy to me because I don't really know what it does to the other columns that I don't want to change (such as datetime columns).
Is there a way to avoid this "upcast" in the first place without affecting other columns?