I am trying to connect to abfss directly(without mounting to DBFS) and trying to open json file using open() in databricks

user_15592807 0
Viewed 1161

I am trying to connect to abfss directly(without mounting to DBFS) and trying to open json file using open() method in databricks.

json_file = open("abfss://@.dfs.core.windows.net/test.json') databricks is unable to open file present in azure blob container and getting below error : FileNotFoundError: [Errno 2] No such file or directory: 'abfss://@.dfs.core.windows.net/test.json'

I have done all the configuration setting using service principal. Please suggest other way of opening file using abfss direct path.

open-json azure-blob-storage pyspark azure-databricks azure
1 Answers

the open method works only with local files - it doesn't know anything about abfss or other cloud storages. You have following choice:

  1. use dbutils.fs.cp to copy file from ADLS to local disk of driver node, and then work with it, like: dbutils.fs.cp("abfss:/....", "file:/tmp/my-copy")
  2. Copy file from ADLS to driver node using the Azure SDK

The first method is easier to use than second

Alex_Ott_186270
Related