Pandas: Create binary columns (dummy/one-hot encode table) based on string column of predefined categories

You could try something like this:

import pandas as pd
import numpy as np

categories = ['RED', 'BLUE', 'YELLOW', 'PINK', 'SHIRT', 'SKIRT']

def categorize(df, categories):
    for category in categories:
        df[category] = np.where(df.Description.str.contains(category), 1, 0)
    return df 

df = categorize(df, categories)

Output:

If you make the items have a uniform separator, you can use the sep parameter of pd.Series.st.get_dummies() to get out the one-hot encoding. I do this by replacing the spaces with commas:

>>> df['Description'].str.replace(' ', ',').str.get_dummies(sep=',')
   BLUE  PINK  RED  SHIRT  SKIRT  YELLOW
0     1     0    1      1      0       0
1     0     0    0      1      0       1
2     1     1    0      0      1       0

Then you just need to join:

>>> df.join(df['Description'].str.replace(' ', ',').str.get_dummies(sep=','))
   Item       Description  BLUE  PINK  RED  SHIRT  SKIRT  YELLOW
0  R2G1  RED, BLUE, SHIRT     1     0    1      1      0       0
1  G23A      YELLOW SHIRT     0     0    0      1      0       1
2  P001  BLUE, PINK SKIRT     1     1    0      0      1       0

Important to note though (as commented by Rob), is that this is determining the categories from the Description column, rather than from the categories list itself. So if you have descriptions that are not in categories, you would have extra columns. E.g. if one description included "GREEN", you would get a whole column for GREEN.

Similarly, categories not present in the descriptions would not be included as columns. So if say the first row was missing, there would be no column for RED.

I can think of ways to fix those behaviors if that is an issue, but I think simpler would be to just go with heretolearn's answer or one of the others that explicitly incorporates categories.

Another approach with findall and MultiLabelBinarizer

from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer()
f = df['Description'].str.findall('|'.join(categories))
out = df.join(pd.DataFrame(mlb.fit_transform(f),columns=mlb.classes_, index=df.index))

A slower but simpler version of above series.str.get_dummies after findall only after joining them:

out = df.join(df['Description'].str.findall('|'.join(categories))
                         .str.join('|').str.get_dummies())

print(out)



   Item       Description  BLUE  PINK  RED  SHIRT  SKIRT  YELLOW
0  R2G1  RED, BLUE, SHIRT     1     0    1      1      0       0
1  G23A      YELLOW SHIRT     0     0    0      1      0       1
2  P001  BLUE, PINK SKIRT     1     1    0      0      1       0

You can create the dummy table with str.get_dummies(). Then, to confine the columns to use only the predefined categories, we use .reindex()

df_dum = (df['Description'].str.replace(r',?\s+', '|', regex=True)
                           .str.get_dummies()
                           .reindex(categories, fill_value=0, axis=1)
         )

Optionally, if you want to change the final column labels from all capital letters to title case strings, you can take the following extra step:

df_dum.columns = df_dum.columns.str.title()

Finally, join back with the original dataframe:

df.join(df_dum)

Result:

   Item       Description  Red  Blue  Yellow  Pink  Shirt  Skirt
0  R2G1  RED, BLUE, SHIRT    1     1       0     0      1      0
1  G23A      YELLOW SHIRT    0     0       1     0      1      0
2  P001  BLUE, PINK SKIRT    0     1       0     1      0      1

You can split the list, use pandas.get_dummies, and take the max grouped by "Item":

df['vals'] = df['Description'].str.split('[,\s]+')
df2 = df.explode('vals')
(pd.concat([df2[['Item', 'Description']],
            pd.get_dummies(df2['vals'])
           ], axis=1)
   .groupby('Item', as_index=False)
   .max()
)

or, shorter alternative:

(pd.concat([df,
            (pd.get_dummies(df['Description'].str.split('[,\s]+').explode())
               .groupby(level=0).max())
           ], axis=1)
)

output:

   Item       Description  BLUE  PINK  RED  SHIRT  SKIRT  YELLOW
0  G23A      YELLOW SHIRT     0     0    0      1      0       1
1  P001  BLUE, PINK SKIRT     1     1    0      0      1       0
2  R2G1  RED, BLUE, SHIRT     1     0    1      1      0       0

• simple case of checking for each category item in the tokenised string
• join() result of this back to dataframe to get your required output

import re

df = pd.DataFrame({'Item': ['R2G1', 'G23A', 'P001'],
 'Description': ['RED, BLUE, SHIRT', 'YELLOW SHIRT', 'BLUE, PINK SKIRT']})

categories = ['RED', 'BLUE', 'YELLOW', 'PINK', 'SHIRT', 'SKIRT']

df.join(df["Description"].apply(lambda s: pd.Series({c:c in re.split("[\, ]", s) for c in categories})).astype(int))