Write CSV file in append mode on Azure Databricks

user_12811183 0
Viewed 789

Wanted to write csv file in append mode on Azure Databricks. The below code is working fine on my local machine.(Jupyter notebook)

df = pd.read_csv("/dbfs/mnt/dev/tmp/ml_p/csv_append.csv")
df+6

Ans [1]: https://i.stack.imgur.com/sXsgH.png

when I opened the same csv file and wanted to save the file after performing the operation. I got , OSError: [Errno 95] Operation not supported

with open('/dbfs/mnt/dev/tmp/ml_p/csv_append.csv', 'a') as f:
   (df + 6).to_csv(f, header=False)

Is there is another alternative to write the CSV file in append mode? or Can I achieve the same using pyspark.

pyspark pandas azure-databricks databricks
1 Answers

There are some limitations on what operations could be done with files on DBFS (especially via /dbfs mount point), and you hit this limit. The workaround would be to copy file from DBFS to local file system, modify it the same as you do it, and then upload back. Copying of the file could be done with dbutils.fs commands, like:

dbutils.fs.cp("dbfs:/mnt/dev/tmp/ml_p/csv_append.csv", "file:/tmp/csv_append.csv")
df = pd.read_csv("/tmp/csv_append.csv")
df+6
with open('/tmp/csv_append.csv', 'a') as f:
   (df + 6).to_csv(f, header=False)
dbutils.fs.mv("file:/tmp/csv_append.csv","dbfs:/mnt/dev/tmp/ml_p/csv_append.csv")
Alex_Ott_186270
Related