SparkFiles - path not found

user_12940312 0
Viewed 139

Please, can you help me with this question below? The image with the error is available in the question.

I use Azure databricks for data engineering. Running the same code in databricks community runs without error, but in Azure returns the error that path was not found. Has anyone been through this situation?
I'm using sparkfiles.

cnae = 'https://servicodados.ibge.gov.br/api/v2/cnae/subclasses'
from pyspark import SparkFiles
spark.sparkContext.addFile(cnae)

cnaeDF = spark.read.option("multiLine", True).option("mode", "PERMISSIVE").json("file://"+SparkFiles.get("subclasses"))

pixel raster: rendered error message & stuff

pyspark
1 Answers

It seems like a bug on runtime 10 as spark.sparkContext.addFile(cnae) add it to local storage:

/local_disk0/spark-f1411c54-0a2e-4138-a0ed-c2e6bbfe5ca4/userFiles-7616de8f-3e03-493c-89e6-50fa1f7324ca/subclasses

but SparkFiles.get("subclasses") want to read it from dbfs storage (I tried to add it all possible ways)...

but when magic command is run:

%sh
cp -r /local_disk0/spark-f1411c54-0a2e-4138-a0ed-c2e6bbfe5ca4 /dbfs/local_disk0/

then it is possible to read it without problem

user_109729590
Related