How to use pandas.get_dummies on prediction time

Viewed 154

I have to convert categorical value columns to run a deep learning model. In training time I am using pd.get_dummies(data=data, dummy_na=True) , and it is doing completely fine.

But In real-time, I have some new categories in each column as expected. So I have to put it ` column_name_nan' for all of that categories. as get_dummies does not do that, I am written it on my own. I have saved the training time columns in a file and used them in prediction time. But it takes a lot of time for big data like 1M. Here is my code.

def populate_data_frame_in_prediction_time(data, columns):
    unknown_col = "nan"
    columns_set = set(columns)
    result_data_frame = pd.DataFrame(0, index=np.arange(len(data)), columns=columns)

    for prefix in data.columns: # O(m)
        unknown_column_name = str(prefix) + "_" + str(unknown_col)
        for index, row in data.iterrows(): #O(n)
            value = row[prefix]
            result_column_name = str(prefix) + "_" + str(value)
            if result_column_name not in columns_set: # O(1)
                result_column_name = unknown_column_name

            result_data_frame[result_column_name][index] = 1

    result_data_frame = result_data_frame.astype('uint8')
    return result_data_frame

do you guys have a better solution for this?

lets say

    training_data = pd.DataFrame({'City': ['Delhi', None, 'Hyderabad', 'Delhi', 'Hyderabad'],
                         'Country': ['D', 'H', 'D', 'H', None]})
    real_time_data = pd.DataFrame({'City': ['Delhi', None, 'Dhaka'],  # })
                                   'Country': [None, "H", "B"]})
    #encoded_columns_in_training_time_to_use_it_on_prediction_time
{'City': ['City_Delhi', 'City_Hyderabad', 'City_nan'], 'Country': ['Country_D', 'Country_H', 'Country_nan']}
2 Answers

For the huge amount of data, you can use dask instead of pandas.

Dask is a Python library similar to Apache Spark in function, but it is closely connected with numpy and pandas, so that python users can learn and use it much more easily than Spark, and it has two functions like Dime.

  1. virtual data frame.
  2. job scheduler for parallel processing

you can get more details Dask Dataframe

Related