Databricks file copy with dbtuils only if file doesn't exist

user_6513537 0
Viewed 10482

I'm using the following databricks utilites (dbutils) command to copy files from one location to another as shown below:

dbutils.fs.cp('adl://dblake.azuredatalakestore.net/jfolder2/thisfile.csv','adl://cadblake.azuredatalakestore.net/landing/')

However, I want the file to be copied over only if no such file with the same name 'thisfile.csv' exists.

Can someone let me know whether is that possible?

If not, is there any other workaround?

azure-databricks databricks
1 Answers

dbutils.fs.ls() lists files in the given path.

ls

So you can check if thisfile.csv exists before copying the file:

if "thisfile.csv" not in [file.name for file in dbutils.fs.ls("adl://cadblake.azuredatalakestore.net/landing/")]:
    dbutils.fs.cp("adl://dblake.azuredatalakestore.net/jfolder2/thisfile.csv", "adl://cadblake.azuredatalakestore.net/landing/")
user_26849540
Related