I'm using Spark MLlib functions (through the sparklyr package) to train a model but now seem unable to save the model in MLflow for future use.
iris_tbl <- sparklyr::copy_to(sc, iris, "iris_spark")
mdl_mllib <- iris_tbl %>% sparklyr::ml_linear_regression(formula = Sepal_Width ~ Sepal_Length)
mlflow::mlflow_log_model(mdl_mllib, "artifact_path_where_saved")
Error in UseMethod("mlflow_save_model") :
no applicable method for 'mlflow_save_model' applied to an object of class "c('ml_model_linear_regression', 'ml_model_regression', 'ml_model_prediction', 'ml_model')"
packageVersion("mlflow")
[1] ‘1.17.0’
What is a simple way to save this model in mlflow for later use on a Spark DataFrame such as:
mlflow::mlflow_load_model(model_uri = "models:/mdl_mllib_project01/Staging")
For context, I'm using Azure Databricks as the ecosystem.
Other places I've looked for answers
- These links don't seem to directly solve my problem (https://www.mlflow.org/docs/latest/models.html#id28; https://www.mlflow.org/docs/latest/python_api/mlflow.spark.html#mlflow.spark.log_model), although maybe it's that I need to make a "pipeline" or that this works for Python but not yet for R?
- This question seems relevant but offers abstract answers rather than details (How to serve a Spark MLlib model?).