type conversion in pandas on assignment of DataFrame series

user_608558 0
Viewed 41

I am noticing something a little strange in pandas (1.4.3). Is this the expected behaviour? The result of an optimization, or a bug? Basically I'd like to guarantee the type does not change unexpectedly, I'd at least like to see an error raised, so any tips are welcome.

If you assign all values of a series in a DataFrame this way, the dtype is altered

>>> import pandas as pd
>>> import numpy as np

>>> df1 = pd.DataFrame({"a": np.array([1,2,3], dtype="int32")})
>>> df1.iloc[:, df1.columns.get_loc("a")] = 0
>>> df1["a"].dtype
dtype('int64')

and if you index the rows in a different way pandas does not convert the dtype

>>> df2 = pd.DataFrame({"a": np.array([1,2,3], dtype="int32")})
>>> df2.iloc[0:len(df2.index), df2.columns.get_loc("a")] = 0
>>> df2["a"].dtype
dtype('int32')
pandas python
3 Answers

Not really an answer but some thoughts that might help you in your quest. My guess as to what is happening is this. In your multiple choice question above I am picking option A - optimization.

I think when 'pandas' sees df1.iloc[:, df1.columns.get_loc("a")] = 0 it is thinking full column(s) replacement of all rows. No slicing - even though df1.iloc[: ... ] is involved. [:] gets translated into all-rows-not-a-slice mode. When it sees = 0 it sees that (via broadcast) as full column(s) of int64. And since it is full replacement then the new column has the same dtype as the source.

But when it sees df2.iloc[0:len(df2.index), df2.columns.get_loc("a")] = 0 it goes into index-slice mode. Even though it is a full-column index slice it doesn't know that and makes an early decision to go into index-slice mode. Index-slice mode then operates on the assumption that only part of the column is going to be updated - not a replacement. Then in update mode the column is assumed to be partially updated and retains its existing dtype.

I got the above hypothesis from looking around at this: https://github.com/pandas-dev/pandas/blob/main/pandas/core/indexes/base.py

If I didn't have a day job I might have the time to actually find the smoking gun in those 6242 lines of code.

user_39687610 
If you look at this code ( I wrote your code little differently to see what is 
happening in the middle)

from pandas._libs import index
import pandas as pd
import numpy as np
dfx= pd.DataFrame({"x": np.array([4,5,6], dtype="int32")}
P=dfx.iloc[:, dfx.columns.get_loc("x")] = 0
P1=dfx.iloc[:, dfx.columns.get_loc("x")]
print(P1)# here you are automatically changing the datatype to int64 ( while 
keep the value 0 , as int64 is default access mechanism for the hardware to 
process the data. 
print(P) 
print(dfx["x"].dtype) 

dfy= pd.DataFrame({"y": np.array([4,5,6], dtype="int32")}) 
Q=dfy.iloc[0:len(dfy.index), dfy.columns.get_loc("y")] = 0 
print(Q) 
Q1=dfy.iloc[0:len(dfy.index), dfy.columns.get_loc("y")] 
print(Q1) 
print(dfy["y"].dtype) 
print(len(dfx.index))
print(len(dfy.index))
user_195466330

Don't know why this is happening, but adding square brackets seem to solve the issue:

df1.iloc[:, [df1.columns.get_loc("a")]] = 0

An other solution seems to be:

df1.iloc[range(len(df1.index)), df1.columns.get_loc("a")] = 0
user_74758380
Related