I am using PySpark on Azure Databricks. I have a pipeline with StringIndexers, OneHotEncoders, a VectorAssembler, and a LinearSVC:
pipeline = Pipeline(stages=["stringvar1_indexer", "stringvar2_indexer", "stringvar2_encoder", "assembler", "svc"])
To find the optimal parameters for the SVC, I use a paramGrid to iterate on regularization parameter and the max iterations:
paramGrid = ParamGridBuilder() \
.addGrid(svc.regParam, [.1, .001, .0001]) \
.addGrid(svc.maxIter, [5, 10, 15]) \
.build()
Then I put it in the TrainValidationSplit to run the SVC on the parameter grid:
tvs = TrainValidationSplit(estimator=pipeline,
estimatorParamMaps=paramGrid,
evaluator=BinaryClassificationEvaluator(),
trainRatio=0.8)
When I run the model with tvs.fit(train), I think it is running the whole indexer, encoder, assembler pipeline each time even though those don't change between runs of the SVC's parameters. Three questions:
- Is it running the whole pipeline each time?
- If so, is there a way for me to cache the non-svc pipeline so it doesn't run each time?
- I am using onnx for model portability, meaning I need to run the whole pipeline together to properly save the model (onnx would not accept vectorUDT inputs when I tried it separately in an earlier iteration). If I have to split the pipeline, can you think of a way to keep onnx so it takes the vector inputs?