Unable to save file in DBFS

Learner1122 0
Viewed 72

enter image description hereI have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :

" No such file or directory: '/dbfs/temp/hive/mytest.csv'"

but on the other hand if I am able to access the path directly from DBFS. This path is correct.

My code :

from azureml.opendatasets import NoaaIsdWeather
from datetime import datetime
from dateutil import parser 
from dateutil.relativedelta import relativedelta


spark.sql('DROP Table if exists mytest')
dbutils.fs.rm("dbfs:/tmp/hive",recurse = True)

basepath = "dbfs:/tmp/hive" 

try:
  dbutils.fs.ls(basepath)
except:
  dbutils.fs.mkdirs(basepath)
else:
  raise Exception("The Folder "+ basepath + " already exist, this notebook will remove in the end")

dbutils.fs.mkdirs("dbfs:/tmp/hive")

start_date = parser.parse('2020-5-1')
end_date = parser.parse('2020-5-10')

isd = NoaaIsdWeather(start_date, end_date)
pdf = isd.to_spark_dataframe().toPandas().to_csv("/dbfs/temp/hive/mytest.csv")

What should I do ?

Thanks

csv databricks apache-spark azure
1 Answers

I tried reproducing the same issue. First I have used the following code and made sure that the directory exists using os.listdir().

from azureml.opendatasets import NoaaIsdWeather
from datetime import datetime
from dateutil import parser 
from dateutil.relativedelta import relativedelta
spark.sql('DROP Table if exists mytest')
dbutils.fs.rm("dbfs:/tmp/hive",recurse = True)
basepath = "dbfs:/tmp/hive" 
try:
  dbutils.fs.ls(basepath)
except:
  dbutils.fs.mkdirs(basepath)
else:
  raise Exception("The Folder "+ basepath + " already exist, this notebook will remove in the end")

dbutils.fs.mkdirs("dbfs:/tmp/hive")

import os  
os.listdir("/dbfs/tmp/hive/")

enter image description here

  • Then I used the following to write the csv using to_pandas_dataframe(). This has successfully written the required dataframe to csv file in required path.
mydf = isd.to_pandas_dataframe()  
mydf.to_csv("/dbfs/tmp/hive/mytest.csv")
BhanunagasaiVamsi0
Related