How to pass a parameter to only one part of a PySpark ML Pipeline object?

Viewed 137

Does anyone know how to pass a parameter to only one step of a PySpark ML Pipeline object? I know that in the case of sklearn it is possible to set parameters of the various steps using their names and the parameter name separated by a ‘__‘. Below is an sklearn example from this SO question How to pass a parameter to only one part of a pipeline object in scikit learn?

m = Pipeline([
    ('feature_selection', SelectKBest(
        score_func=sklearn.feature_selection.f_regression,
        k=25)),
    ('model', RandomForestClassifier(
        random_state=0, 
        oob_score=True, 
        n_estimators=500,
        min_samples_leaf=5, 
        max_depth=10))])

m.fit(X, y, model__sample_weight=np.array([3,4,2,3]))

I am looking for a similar solution in PySpark. Let's say this is my pipeline:

tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
lr = LogisticRegression(maxIter=10, regParam=0.001)
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])
0 Answers
Related