ValueError when loading a PipelineModel on a personal computer

user_8947333 0
Viewed 187

I have a PipelineModel saved on my computer that I can't load using PipelineModel.load(path).

When I launch my code in a Databricks cluster, it works. path is the path to my model saved on DBFS, accessible via a mount point: path = "/dbfs/path/to/my/model.

However on my machine, calling PipelineModel.load("C:\\Users\\path\\to\\my\\model") throws a ValueError("RDD is empty").

Here is how the model is saved on my computer:

\---model
    +---metadata
    |       part-00000
    |       _SUCCESS
    |
    \---stages
        +---0_CountVectorizer_b92625354bf7
        |   +---data
        |   |       part-00000-tid-9156766819779394023-5cf6aecb-8959-48b3-be24-65bfa0543465-62-1-c000.snappy.parquet
        |   |       _committed_9156766819779394023
        |   |       _started_9156766819779394023
        |   |       _SUCCESS
        |   |
        |   \---metadata
        |           part-00000
        |           _SUCCESS
        |
        \---1_LinearSVC_108fa01daf43
            +---data
            |       part-00000-tid-4403060754466700849-27841dd9-de88-4015-9dfa-7854c2a15f15-65-1-c000.snappy.parquet
            |       _committed_4403060754466700849
            |       _started_4403060754466700849
            |       _SUCCESS
            |
            \---metadata
                    part-00000
                    _SUCCESS

(I just downloaded the model from my DataLake to my computer)

How can I load this model when running my code in local?

pyspark apache-spark python
1 Answers

For others struggling to solve the same problem, be sure to actually download the blob... I had a typo in my code that caused the wrong blob to be downloaded, leading to all files being empty.

user_89473330
Related