Pandas .map dictionary default missing value

Question

Pandas Docs for pandas.Series.map says that:

"When arg is a dictionary, values in Series that are not in the dictionary (as keys) are converted to NaN. However, if the dictionary is a dict subclass that defines missing (i.e. provides a method for default values), then this default is used rather than NaN."

How do you actually do that? I cannot get it to work..

class MyDict(collections.UserDict):
    
  def __missing__(self):
    return "_Unknown"

d = MyDict({k: v for k, v in my_list})

df.col1.map(d)

user_4333359 · Answer

You need (self, key) as the arguments for __missing__:

class MyDict(dict):
    def __missing__(self, key):
        return "_Unknown"

import pandas as pd

s = pd.Series(range(4))
d = {0: 'foo', 1: 'bar', 2: 'baz'}

s.map(MyDict(d))
#0         foo
#1         bar
#2         baz
#3    _Unknown
#dtype: object

And while this functionality is nice, map is a very efficient pandas method when it uses standard dictionaries, the above will slow it down (shown below). So instead you can map with a normal dict then chain on a .fillna to handle the default.

s.map(d).fillna('_Unknown')
#0         foo
#1         bar
#2         baz
#3    _Unknown
#dtype: object

Illustrations of the timings using a dictionary with a special return for missing keys versus a normal dictionary followed by a .fillna. MyDict is a lot faster for very small mappings, but is slower for larger Series.

import perfplot
import pandas as pd
import numpy as np

class MyDict(dict):
    def __missing__(self, key):
        return "_Unknown"

d = {0: 'foo', 1: 'bar', 2: 'baz'}  
d2 = MyDict(d)

def map_fillna(s, d):
    return s.map(d).fillna("_Unknown")

def use_MyDict(s, MyDict):
    return s.map(MyDict)


perfplot.show(
    setup=lambda n: pd.Series(np.random.choice(range(7), n)), 
    kernels=[
        lambda s: map_fillna(s, d),
        lambda s: use_MyDict(s, d2),
    ],
    labels=['map + fillna', 'MyDict'],
    n_range=[2 ** k for k in range(1, 27)],
    equality_check= lambda x,y: x.compare(y).empty,
    xlabel='len(s)'
)

Pandas .map dictionary default missing value

1 Answers