Does using paramGrid with TrainValidationSplit cause the entire pipeline to run each time or just the step that needs to be optimized?

Viewed 63

I am using PySpark on Azure Databricks. I have a pipeline with StringIndexers, OneHotEncoders, a VectorAssembler, and a LinearSVC:

pipeline = Pipeline(stages=["stringvar1_indexer", "stringvar2_indexer", "stringvar2_encoder", "assembler", "svc"])

To find the optimal parameters for the SVC, I use a paramGrid to iterate on regularization parameter and the max iterations:

paramGrid = ParamGridBuilder() \
    .addGrid(svc.regParam, [.1, .001, .0001]) \
    .addGrid(svc.maxIter, [5, 10, 15]) \
    .build()

Then I put it in the TrainValidationSplit to run the SVC on the parameter grid:

tvs = TrainValidationSplit(estimator=pipeline,
    estimatorParamMaps=paramGrid,
    evaluator=BinaryClassificationEvaluator(),
    trainRatio=0.8)

When I run the model with tvs.fit(train), I think it is running the whole indexer, encoder, assembler pipeline each time even though those don't change between runs of the SVC's parameters. Three questions:

  1. Is it running the whole pipeline each time?
  2. If so, is there a way for me to cache the non-svc pipeline so it doesn't run each time?
  3. I am using onnx for model portability, meaning I need to run the whole pipeline together to properly save the model (onnx would not accept vectorUDT inputs when I tried it separately in an earlier iteration). If I have to split the pipeline, can you think of a way to keep onnx so it takes the vector inputs?
0 Answers
Related