I have to convert categorical value columns to run a deep learning model. In training time I am using pd.get_dummies(data=data, dummy_na=True) , and it is doing completely fine.
But In real-time, I have some new categories in each column as expected. So I have to put it ` column_name_nan' for all of that categories. as get_dummies does not do that, I am written it on my own. I have saved the training time columns in a file and used them in prediction time. But it takes a lot of time for big data like 1M. Here is my code.
def populate_data_frame_in_prediction_time(data, columns):
unknown_col = "nan"
columns_set = set(columns)
result_data_frame = pd.DataFrame(0, index=np.arange(len(data)), columns=columns)
for prefix in data.columns: # O(m)
unknown_column_name = str(prefix) + "_" + str(unknown_col)
for index, row in data.iterrows(): #O(n)
value = row[prefix]
result_column_name = str(prefix) + "_" + str(value)
if result_column_name not in columns_set: # O(1)
result_column_name = unknown_column_name
result_data_frame[result_column_name][index] = 1
result_data_frame = result_data_frame.astype('uint8')
return result_data_frame
do you guys have a better solution for this?
lets say
training_data = pd.DataFrame({'City': ['Delhi', None, 'Hyderabad', 'Delhi', 'Hyderabad'],
'Country': ['D', 'H', 'D', 'H', None]})
real_time_data = pd.DataFrame({'City': ['Delhi', None, 'Dhaka'], # })
'Country': [None, "H", "B"]})
#encoded_columns_in_training_time_to_use_it_on_prediction_time
{'City': ['City_Delhi', 'City_Hyderabad', 'City_nan'], 'Country': ['Country_D', 'Country_H', 'Country_nan']}