Pandas replace does not work with different numpy-int types even when values are the same

user_6224975 0
Viewed 65

I came across a strange problem today while debugging. Long story short, I have a clustering model which yields some labels to each observation. I want to change these labels according to their clustering-mean, such that the cluster with the highest mean has the biggest cluster-number and so on.

The issue is that my mapping dictionary-keys have np.int64 as type but my series of labels have np.int32 as type which causes the replacement not to happen (which I find very odd!)

See this example:

maps = pd.Series([np.int64(0),np.int64(2),np.int64(1)]) #index-value should be mapped to series-value

# 0 0    0-->0
# 1 2    1-->2
# 2 1    2-->1

map_dict = {old:new for (old,new) in zip(maps.values,maps.index)}

map_dict == {0: 0, 2: 1, 1: 2} #True

labs = pd.Series([1,1,1,0,0,2,2,2]).astype(np.int32) #Simulate my list of clusters of observations

(labs.replace({0: 0, 2: 1, 1: 2}) == labs.replace(map_dict)).mean() #0.25

As you can see the map_dict=={0: 0, 2: 1, 1: 2} evalutates to True thus I would assume using either of those produces the same result.

Further more, this is odd:

np.int64(1) == np.int(32) # True
1 == np.int(32) # True

i.e it shouldn't make a difference if I'm using np.int64, np.int32 or just int. Actually, if I convert the map_dict to int like this:

.
.


map_dict = {int(old):int(new) for (old,new) in zip(maps.values,maps.index)} #cast to "int"

map_dict == {0: 0, 2: 1, 1: 2} #True

labs = pd.Series([1,1,1,0,0,2,2,2]).astype(np.int32) #Simulate my list of clusters of observations

(labs.replace({0: 0, 2: 1, 1: 2}) == labs.replace(map_dict)).mean() #1.0 - all of them gets replaced correctly

I'm aware that there technically is a difference between int64 and int32 (the number of bits) but I find the above very strange, that the two mapping dictionaries evaluate to True when we compare them but for some reason we have to use the exact same integer-type when using the replace method.

Also ususally the integer-type doesn't matter (I haven't had such a problem in Python before):

pd.Series([np.int32(1)]) == pd.Series([np.int64(1)]) #True
pandas integer numpy python
1 Answers

The problem is probably the key type of map_dict and the replace function:

maps = pd.Series([np.int64(0),np.int64(2),np.int64(1)])

map_dict = {old:new for (old,new) in zip(maps.values,maps.index)}
for k in map_dict:
    print(type(k))

# <class 'numpy.int64'>
# <class 'numpy.int64'>
# <class 'numpy.int64'>

map_dict = maps.to_dict()
for k in map_dict:
    print(type(k))

# <class 'int'>
# <class 'int'>
# <class 'int'>

Use replace:

# with map_dict generate from comprehension
>>> labs.replace(map_dict)
0    1
1    1
2    1
3    0
4    0
5    2
6    2
7    2
dtype: int32

# with map_dict generate from to_dict
>>> labs.replace(map_dict)
0    2
1    2
2    2
3    0
4    0
5    1
6    1
7    1
dtype: int64

However when you use map instead of replace there is no difference on how the map_dict is built:

>>> labs.map(map_dict)
0    2
1    2
2    2
3    0
4    0
5    1
6    1
7    1
dtype: int64  # <- Note the cast from int32 to int64
user_152399510
Related