How can I OneHot Encode data from multiple files

user_7377486 0
Viewed 153

I have two files train.csv and test.csv, I am trying to clean the datasets and apply feature engineering on both of them separately. Suppose in train.csv I have a column dummy which consists of only two type of values ('A','B'), but in test.csv that same columns contains three type of values ('A','B','C') the value 'C' is missing in train.csv, so now when I will Hot Encode train.csv using pandas dummy method:

data = pd.get_dummies(data,columns=['dummy'],prefix=['dummy'])

I will end up having different number of columns in train and test datasets. This problem occurs in multiple columns.

train.csv:
-------
|dummy |
--------
|A     |
--------
|B     |
--------
|A     |
--------
|A     |
--------


test.csv:
-------
|dummy |
--------
|A     |
--------
|B     |
--------
|C     |
--------
|A     |
--------
one-hot-encoding pandas python
2 Answers

If you know the levels to expect, encode your column as a category. For example:

traindf = pd.DataFrame({'dummy':['A','B','A','A']})
testdf = pd.DataFrame({'dummy':['A','B','C','A']})

lvls = pd.concat([traindf['dummy'],testdf['dummy']]).unique()

traindf['dummy'] = pd.Categorical(traindf['dummy'],categories=lvls)
testdf['dummy'] = pd.Categorical(testdf['dummy'],categories=lvls)


pd.get_dummies(traindf)
 
   dummy_A  dummy_B  dummy_C
0        1        0        0
1        0        1        0
2        1        0        0
3        1        0        0

pd.get_dummies(testdf)
 
   dummy_A  dummy_B  dummy_C
0        1        0        0
1        0        1        0
2        0        0        1
3        1        0        0
user_122584590

I have had this same problem myself, the way I solved it was to store the list of the generated columns after encoding the train set in one list, then after encoding the test set I would filter the new data in order to keep only the columns that were present in the encoded train set. Also, the following function takes care of possible encoded values that are present in the train set but not in the test set:

def encode(X: pd.DataFrame, columns: list = None) -> pd.DataFrame:
    if columns is None:
        columns = []
    encoded_df = pd.get_dummies(X)
    encoded_df = encoded_df.reindex(columns=columns, fill_value=np.uint8(0))
    new_cols = [col for col in list(encoded_df.columns) if col not in columns]
    encoded_df.drop(new_cols, axis=1, inplace=True)
    return encoded_df

You can use this function in the following way:

encoded_train_df = encode(train_df)
train_cols = list(encoded_train_df.columns)
encoded_test_df = encode(test_df, columns=train_cols)
user_110500000
Related