Data type handling in pandas apply() vs applymap()

user_13329006 0
Viewed 122

I observed a different behavior in the handling of data types for the .apply() and the .applymap() functions of pandas.

big_number = 999999

big_number * big_number
Out[715]: 999998000001

np.square(big_number)
Out[716]: -729379967

np.square(float(big_number))
Out[717]: 999998000001.0

df = pd.DataFrame([big_number], columns=["big_number"])

df.dtypes
Out[719]: 
big_number    int64
dtype: object

df.apply(np.square)
Out[720]: 
     big_number
0  999998000001

df.applymap(np.square)
Out[721]: 
   big_number
0  -729379967

df.astype({"big_number" : float}).applymap(np.square)
Out[722]: 
     big_number
0  9.999980e+11

Obviously behaves .applymap() more correctly then .apply(), because it actually reproduces the result of the np.square() function. .apply() seems to do some magic data type conversion before calling the np.square function.

Can anybody explain what happens "under the hood"?

pandas numpy python types
1 Answers

Explanation

Everything boils down to numpy and pandas converting the built-in int type differently.

Under the hood the applymap function applies np.square directly to dataframe entries, so it eventually passes through np.square(big_number). Here big_number is converted by numpy to np.int32 and the result is also 32-bit, so you get -729379967.

On the other hand, the apply function applies np.square to df.values. This is also a numpy's ndarray, but the tricky part is that while creating the DataFrame big_number's value was first interpreted by pandas as 64-bit, hence df.values.dtype is np.int64 and not np.int32. Consequently, also the result is 64-bit and so it's 999998000001.

Additional example

You may observe the same behavior performing the following verification:

np_array = np.array(big_number)
np_array_2 = df.values
print(type(big_number), np_array.dtype, np_array_2.dtype)
print(np.square(np_array), np.square(np_array_2))

You should see the types are int, int32 and int64 and the results are -729379967 and [[999998000001]].

Important

Note that I was able to reproduce your problem only in numpy's 32-bit distribution, but not in the 64-bit one. I guess the int conversion is governed by np.int_, which in the former environment coincides with np.int32, while in the latter one with np.int64.

user_100926380
Related