pandas category: keep only most common ones and replace rest with NaN

Question

In a pd.Series with dtype=category I have 253 unique values. Some of these occur quite often and others occur only once or twice. Now I would like to keep only the top 10 of these and replace the rest with np.nan.

I got as far as top = df['cats'].value_counts().head(10) to create the categories I want to keep. But now?

Something along the lines of df['cats'].apply(cat_replace, args=top)?

def cat_replace(c, top):
    if c in top:
        return c
    else:
        return np.nan

This however doesn't look too 'pandas' to me and I have a feeling there is a better way. Any better suggestions?

user_2411802 · Answer

# Sample data.
df = pd.DataFrame(
    {'cats': pd.Categorical(
        list('abcdefghij') * 5
        + list('klmnopqrstuvwxyz'))}
)

top_n = 10
top_cats = df['cats'].value_counts().head(top_n).index.tolist()
df.loc[~df['cats'].isin(top_cats), 'cats'] = np.nan

user_2284490 · Answer

Cribbing from

How can I keep the rows of a pandas data frame that match a particular condition using value_counts() on multiple columns

You could look at doing something like

top = set(df['cats'].value_counts().head(10))
df['cats'].apply(top.__contains__)

pandas category: keep only most common ones and replace rest with NaN

2 Answers