How to store scaling parameters for later use

user_4068678 0
Viewed 13571

I want to apply the scaling sklearn.preprocessing.scale module that scikit-learn offers for centering a dataset that I will use to train an svm classifier.

How can I then store the standardization parameters so that I can also apply them to the data that I want to classify?

I know I can use the standarScaler but can I somehow serialize it to a file so that I wont have to fit it to my data every time I want to run the classifier?

standardized scikit-learn normalization python
4 Answers

Scale with standard scaler

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(data)
scaled_data = scaler.transform(data)

save mean_ and var_ for later use

means = scaler.mean_ 
vars = scaler.var_

(you can print and copy paste means and vars or save to disk with np.save....)

Later use of saved parameters 

def scale_data(array,means=means,stds=vars **0.5):
    return (array-means)/stds

scale_new_data = scale_data(new_data)
user_85636490

You can use the joblib module to store the parameters of your scaler.

from joblib import dump
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(data)
dump(scaler, 'scaler_filename.joblib')

Later you can load the scaler.

from joblib import load
scaler = load('scaler_filename.joblib')
transformed_data = scaler.transform(new_data)
user_90943790

Pickle brings a security vulnerability and allows attackers to execute arbitrary code on the servers. The conditions are:

  • possibility to replace the pickle file with another pickle file on the server (if no auditing of the pickle performed, i.e. signature validation or hash comparison)

  • the same but on the developer PC (attacker compromised some dev PC

If your server-side applications are executed as root (or under root in docker containers), then this is definitely worth of your attention.

Possible solution:

  • Model training should be done in a secure environment

  • Trained models should be signed by the key from another secure environment, which is not loaded to the gpg-agent (otherwise the attacker can quite easily replace the signature)

  • CI should test the models in an isolated environment (quarantine)

  • Use python3.8 or later which added security hooks to prevent code injection techniques

  • or just avoid pickle:)

Some links:

Possible approach to avoid pickling:

# scaler is fitted instance of MinMaxScaler
scaler_data_ = np.array([scaler.data_min_, scaler.data_max_])
np.save("my_scaler.npy", allow_pickle=False, scaler_data_)

#some not scaled X
Xreal = np.array([1.9261148646249848, 0.7327923702472628, 118, 1083])

scaler_data_ = np.load("my_scaler.npy")
Xmin, Xmax = scaler_data_[0], scaler_data_[1]
Xscaled = (Xreal - Xmin) / (Xmax-Xmin)
Xscaled
# -> array([0.63062502, 0.35320565, 0.15144766, 0.69116555])
user_43510270
Related