Does anyone know how to pass a parameter to only one step of a PySpark ML Pipeline object? I know that in the case of sklearn it is possible to set parameters of the various steps using their names and the parameter name separated by a ‘__‘. Below is an sklearn example from this SO question How to pass a parameter to only one part of a pipeline object in scikit learn?
m = Pipeline([
('feature_selection', SelectKBest(
score_func=sklearn.feature_selection.f_regression,
k=25)),
('model', RandomForestClassifier(
random_state=0,
oob_score=True,
n_estimators=500,
min_samples_leaf=5,
max_depth=10))])
m.fit(X, y, model__sample_weight=np.array([3,4,2,3]))
I am looking for a similar solution in PySpark. Let's say this is my pipeline:
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
lr = LogisticRegression(maxIter=10, regParam=0.001)
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])