How to standardize scikit learn by group?

user_239923 0
Viewed 464

I want to apply the StandardScaler group by another column.

E.g. I want to standardize column1 but grouped by grp1 so that within each distinct value of grp1 a different mean and standard deviation is used.

Is it possible with scikit learn? Or do I need to construct a pipeline for each group separately?

Non scikit learn solutions welcome

scikit-learn python
2 Answers

There is no built-in solution to perform this operation. However, you can create your own Transformer that perform this operation.

The following custom transformer is performing a fitting one StandardScaler for each unique values defined in the by columns. This code expects to receive a DataFrame as input.

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import StandardScaler

import pandas as pd



class GroupByScaler(BaseEstimator, TransformerMixin):
    def __init__(self, by=None):
        self.scalers = dict()
        self.by = by
    
    def fit(self, X, y = None):
        self.cols = df.select_dtypes(exclude=['object']).columns
        for val in X[self.by].unique():
            mask = X[self.by]==val
            X_sub = X.loc[mask, self.cols]
            self.scalers[val] = StandardScaler().fit(X_sub)
        return self
    
    def transform(self, X, y = None):
        for val in X[self.by].unique():
            mask = X[self.by]==val
            X.loc[mask, self.cols] = self.scalers[val].transform(X.loc[mask, self.cols])
            
        return X

The following code is testing that the scaling is different for the group X and Y from column A.

df = pd.DataFrame({'A': ['X', 'X', 'X', 'X', 'Y','Y','Y','Y'], 
                  'B': [10, 10, 10, 10, 0, 1, 2, 4],
                  'C': [0, 1, 2, 4, 10, 10, 10, 10]})
GroupByScaler(by='A').fit_transform(df)

This returns the following output:

   A         B         C
0  X  0.000000 -1.183216
1  X  0.000000 -0.507093
2  X  0.000000  0.169031
3  X  0.000000  1.521278
4  Y -1.183216  0.000000
5  Y -0.507093  0.000000
6  Y  0.169031  0.000000
7  Y  1.521278  0.000000
user_45746330

The solution by Antoine worked for me. I think the first line in the fit method should be:

self.cols = X.select_dtypes(exclude=['object']).columns

and not

self.cols = df.select_dtypes(exclude=['object']).columns

The latter only worked in Antoine's example because his data happened to be named "df".

user_111548010
Related