Why does Pyspark throw : " AnalysisException: `/path/to/adls/mounted/interim_data.delta` is not a Delta table ". even though the file exists...?

user_4010631 0
Viewed 4565

I am using databricks on azure, Pyspark reads data that's dumped in azure data lake storage [adls] Every now and then when i try to read the data from adls like so:
spark.read.format('delta').load(`/path/to/adls/mounted/interim_data.delta` )

it throws the following error

AnalysisException: `/path/to/adls/mounted/interim_data.delta` is not a Delta table.

the data necessarily exists
the folder contents and files show up when i run
%fs ls /path/to/adls/mounted/interim_data.delta

right now the only fix is to re run the script that populated the above interim_data.delta table which is not a viable fix

azure-data-lake-gen2 delta-lake pyspark azure-databricks
2 Answers

Make sure you have copied the data in delta format correctly.

Below is the standard command to do so:

df = spark.read.format(file_type).option("header","true").option("inferSchema", "true").option("delimiter", '|').load(file_location)

df.write.format("delta").save()

You access data in Delta tables either by specifying the path on DBFS ("/mnt/delta/events") or the table name ("events"). Make sure the path or file name should be in correct format. Please refer example below:

val events = spark.read.format("delta").load("/mnt/delta/events")

Refer https://docs.microsoft.com/en-us/azure/databricks/delta/quick-start#read-a-table to know more about Delta Lake.

Feel free to ask in case you have any query.

user_159690410

I am answering my own question...

TLDR: Root cause of the issue: frequent remounting of ADLS

There was this section of the code that remounts the ADLS gen2 to ADB, when other teams ran their script, the remounting took 20-45 seconds, and as the number of scripts that ran in the high concurrency cluster increased, it was a matter of time that one of us ran into the issue, where the scripts tired to read data from the ADLS while it was being mounted...

this is how it turned out to be intermittent...

Why was this remounting hack in place..? this was put in place because, we faced an issue with data not showing up, in adb even though it was visible in ADLS Gen2, and the only way to fix this back then, was to force a remount to make that data visible in ADB.

user_40106310
Related