MLflow FileNotFound when calling spark_udf from pycharm with databricks-connect

I am saving a scikit-learn trained model to MLflow's model registry in my Windows laptop. I am using databricks-connect to connect to an Azure Databricks cluster and train models there from my local Pycharm, but for development I launch the model registry in my laptopt and save the trained models in it - to avoid having to set up accessing DBFS remotely.

I am having trouble with the spark_udf function. I can read the model from my laptop's model registry but cannot use it as a UDF in my Databricks cluster:

model_udf = mlflow.pyfunc.spark_udf(spark, "models:/mymodel/production")  # this works fine
struct_col = F.struct(*df.columns)
predictions = df.withColumn("pred_spark", model_udf(struct_col))  
predictions.show()                                        # throws an exception(see below)

pyspark.sql.utils.PythonException: An exception was thrown from a UDF: 'FileNotFoundError:

[Errno 2] No such file or directory:

'/local_disk0/spark-1fa39b20-9d2c-4697-957c-392d80326dee/executor-57b039d8-7405-47c4-b072-612e9b87b3dd/spark-e442241d-4007-4c6e-8acd-bf2a35b1a455/isolatedSparkFiles/044cd765-f5f7-46b3-9efb-0944cc91ef4d/c:\temp\tmpsl4hpeyt.zip'

The last part is weird as it mixes linux-style routes with Windows route (like a route in my Windows laptop's local dir). I thought that the driver would read the model from the model registry and broadcast it to the workers to call the UDF, but looks like the workers are trying to fetch it directly from the remote model registry, is this right? Is there a solution that does not require saving the model in the remote model registry - or at least, configuring the security to access DBFS?

EDIT: After having set up registering the model into the remote registry (not that difficult!) I can now download the registered model as a sklearn model and do predictions with it, but I cannot do the same via spark_udf. I get either

• The same FileNotFound error if I specify the model via runs:/.... in function spark_udf
• An SSL error (unverified self-signed certificate) if I specify the model via model:/... in the spark_udf, even though I have explicitly added a new line insecure = True in my .databrickscfg - and because that was not working, also added os.environ["MLFLOW_TRACKING_INSECURE_TLS"] = "true" but has no effect. I guess it is doing something because at least I can download the model from the registry when it is not a spark_udf, but looks like "the workers?" are having issues when they try to do the same (that's my guess, no idea if it makes sense)