Find pandas dataframe columns which are considered as floats but actually can be written as integers

user_9811964 0
Viewed 92

I have a dataframe with multiple columns.

df = pd.DataFrame({'A' : [1.0, 3.0, 4.0, 5.0, 2.0], 
                  'B' : [2, 4, 5, 8, 9],
                  'C' : [1.8, 4.1, 4.0, 5.6, 2.0],
                  'D' : [99, 100, 101, 101, 99],
                  'D' : [99.0, 1000.0, np.nan, 101.0, 99.0]})
df

     A   B   C    D
0   1.0  2  1.8  99.0
1   3.0  4  4.1  1000.0
2   4.0  5  4.0  NaN
3   5.0  8  5.6  101.0
4   2.0  9  2.0  99.0

After applying dtype, we see that columns A and D are considered as float.

df.dtypes

A    float64
B      int64
C    float64
D    float64
dtype: object

I want to find all columns in my df, which can be represented as integers but are considered as floats.

Expected result:

['A', 'D']

The list contains all columns, which are considered as floats but actually can be represented as integers.

How can I find these columns?

pandas dataframe format integer
3 Answers

We can use DataFrame.convert_dtypes to check which columns are converted to Int64 but were float64 in the original dataframe:

dtypes_old = df.dtypes
dtypes_new = df.convert_dtypes().dtypes

s = dtypes_old.eq("float64") & dtypes_new.eq("Int64")
s[s].index

Index(['A', 'D'], dtype='object')

Note that Int64 is a Nullable integer dtype and is an pandas extension dtype.

user_90812670

Use Index.intersection of float and Int64 columns created by DataFrame.convert_dtypes:

c1 = df.select_dtypes(np.float).columns
c2 = df.convert_dtypes().select_dtypes('Int64').columns

out = c1.intersection(c2, sort=False).tolist()
print (out)
['A', 'D']
user_29010020

One way to do it via df.convert_dtypes():

pd.concat([df.dtypes,df.convert_dtypes().dtypes]).astype(str).str.title().reset_index().drop_duplicates(keep=False)['index'].unique()

OUTPUT:

array(['A', 'D'], dtype=object)
user_154380330
Related