I am currently working on a Machine Learning model and I am in the process of feature engineering. I am using a dataset that indicates the number of product sales of 70+ items across 13 stores.
I have created features such as the average price of an item based on its SKU-ID (what product it is). I now wish to make a feature of the standard deviation for each SKU price.
Following is the function I used for generating the average price of a particular product across all stores as well as my function for determining the standard deviation of a product's price across all stores:
def gen_average_price(train,test,col,price='base_price',name='name'):
temp=train.groupby(col)[price].mean().reset_index().rename(columns={price:name})
train=pd.merge(train,temp,how='left',on=col)
test=pd.merge(test,temp,how='left',on=col)
train[name].fillna(np.median(temp[name]),inplace=True)
test[name].fillna(np.median(temp[name]),inplace=True)
return train,test
def gen_std_price(train,test,col,price='base_price',name='name'):
temp=train.groupby(col)[price].mean().reset_index().rename(columns={price:name})
train=pd.merge(train,temp,how='left',on=col)
test=pd.merge(test,temp,how='left',on=col)
s = np.median(temp[name])
train[name].fillna(statistics.stdev(s), inplace = True)
test[name].fillna(statistics.stdev(s), inplace = True)
return train,test
This is the function i use to obtain the average price for a specific product as well as the funtion to obtain the standard deviation of a product's price. The average price function (gen_average_price) above works as intended, however when i attempt to use my standard deviation function for obtaining the standard deviation of a product's price i run in to the following error:
#Generating std dev of prices per sku-id
train,test = gen_std_price(train,test,col=['sku_id'],price='base_price',name='price_sku_id_stdev')
TypeError: 'numpy.float64' object is not iterable
Can anyone help me as to why this is the case and/or how to fix it?
Following is some more info on the datasets used:
train.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 150150 entries, 0 to 150149
Data columns (total 16 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 record_ID 150150 non-null int64
1 week 150150 non-null object
2 store_id 150150 non-null int64
3 sku_id 150150 non-null int64
4 total_price 150150 non-null float64
5 base_price 150150 non-null float64
6 is_featured_sku 150150 non-null int64
7 is_display_sku 150150 non-null int64
8 units_sold 150150 non-null int64
9 count_id_sku_store 150150 non-null float64
10 count_id_sku 150150 non-null float64
11 count_id_store 150150 non-null float64
12 price_sku_store 150150 non-null float64
13 price_to_sku_store 150150 non-null float64
14 price_store_id 150150 non-null float64
15 price_sku_id 150150 non-null float64
dtypes: float64(9), int64(6), object(1)
memory usage: 19.5+ MB
test.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 13860 entries, 0 to 13859
Data columns (total 15 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 record_ID 13860 non-null int64
1 week 13860 non-null object
2 store_id 13860 non-null int64
3 sku_id 13860 non-null int64
4 total_price 13860 non-null float64
5 base_price 13860 non-null float64
6 is_featured_sku 13860 non-null int64
7 is_display_sku 13860 non-null int64
8 count_id_sku_store 13860 non-null float64
9 count_id_sku 13860 non-null float64
10 count_id_store 13860 non-null float64
11 price_sku_store 13860 non-null float64
12 price_to_sku_store 13860 non-null float64
13 price_store_id 13860 non-null float64
14 price_sku_id 13860 non-null float64
dtypes: float64(9), int64(5), object(1)
memory usage: 1.7+ MB
