Using Keras StandardScaler with Groupby function in Pandas

user_14011227 0
Viewed 396

I have a pandas data frame with multiple columns. I need to use groupby function on each column and after use Keras StandardScaler function to transform each column in the dataframe. I tried the following code:

from sklearn.preprocessing import StandardScaler
scaler=StandardScaler()
df2= df.groupby('Sector').apply(lambda x: scaler.fit_transform(x.astype(float)))

but it returns lists of data by group, however, I need to preserve the initial structure of the dataframe.

I specifically need to use StandardScaler because afterwards I want to use it to transform testing features.

Is there a way to use StandardScaler in this case?

pandas keras python-3.x python
2 Answers

I don't understand why you are using group by.

To get a copy of data

scaled_features = data.copy()

To scale only a few columns

features = scaled_features[['column1','column2']]
scaler = StandardScaler().fit(features.values)
features = scaler.transform(features.values)

To transform columns also you can do it in your way. Or also using a query you can get only these data where the value matches your query.

df_c = df['column'].apply(lambda x: scaler.fit_transform(x.astype(float)))
df_cc = df[df['column'] == '...'].apply((lambda x: scaler.fit_transform(x.astype(float)))

Another way is to use ColumnTransformer

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler

ct = ColumnTransformer([
        ('name', StandardScaler(), ['column1', 'column2'])
    ], remainder='passthrough')

ct.fit_transform(features)
user_51435540

I think there is no such option in the StandardScaler library but following pandas commands can be of help.

# Normalize around mean
df['normal'] = df.groupby('Group').transform(lambda x: (x - x.mean()/ x.std()))
# Normalize between 0 and 1
df['normal'] = df.groupby('Group').transform(lambda x: ((x - x.min())/ (x.max() - x.min())))

Regarding testing data, I think you should do the normalization before and then split it into train and test.

user_59284360
Related