How to move files from one folder to another on databricks

user_13950035 0
Viewed 9589

I am trying to move the file from one folder to another folder using databricks python notebook. My source is azure data lake gen 1.

Suppose, my file is present adl://testdatalakegen12021.azuredatalakestore.net/source/test.csv and I am trying to move the file from adl://testdatalakegen12021.azuredatalakestore.net/demo/test.csv to adl://testdatalakegen12021.azuredatalakestore.net/destination/movedtest.csv

I tried various logic but not none of my code is working fine.

# Move a file by renaming it's path
import os
import shutil
os.rename('adl://testdatalakegen12021.azuredatalakestore.net/demo/test.csv', 'adl://testdatalakegen12021.azuredatalakestore.net/demo/renamedtest.csv')

# Move a file from the directory d1 to d2
shutil.move('adl://testdatalakegen12021.azuredatalakestore.net/demo/test.csv', 'adl://testdatalakegen12021.azuredatalakestore.net/destination/renamedtest.csv')

Please, let me know If I am using correct logic as I am executing this on databricks, not in my local.

azure-data-lake pyspark azure-databricks apache-spark python
3 Answers

To move a file in databricks notebook, you can use dbutils as follow:

dbutils.fs.mv('adl://testdatalakegen12021.azuredatalakestore.net/demo/test.csv', 'adl://testdatalakegen12021.azuredatalakestore.net/destination/renamedtest.csv')
user_68077690

Here are the steps to move files from one folder to another on databricks:

Mount the Azure Data Lake Storage Gen1 to the databricks workspace:

configs = {"<prefix>.oauth2.access.token.provider.type": "ClientCredential",
           "<prefix>.oauth2.client.id": "<application-id>",
           "<prefix>.oauth2.credential": dbutils.secrets.get(scope = "<scope-name>", key = "<key-name-for-service-credential>"),
           "<prefix>.oauth2.refresh.url": "https://login.microsoftonline.com/<directory-id>/oauth2/token"}

# Optionally, you can add <directory-name> to the source URI of your mount point.
dbutils.fs.mount(
  source = "adl://<storage-resource>.azuredatalakestore.net/<directory-name>",
  mount_point = "/mnt/<mount-name>",
  extra_configs = configs)

enter image description here

Reference: Mount Azure Data Lake Storage Gen1 resource using a service principal and OAuth 2.0

Moving file using %fs command

%fs mv dbfs:/mnt/adlsgen1/test/mapreduce.txt dbfs:/mnt/adlsgen1/test1/mapreduce.txt

enter image description here

Moving file using dbutils command:

dbutils.fs.mv('dbfs:/mnt/adlsgen1/test/data.csv', 'dbfs:/mnt/adlsgen1/test1/dataone.csv')

enter image description here

user_81884330

you may want to also move/or copy the file without their subdirectories

import os

source_dir = "/mnt/yourplateform/source"
dest_dir= "/mnt/yourplateform/destination//"

list_of_files=[]
fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration())
path_exists = fs.exists(spark._jvm.org.apache.hadoop.fs.Path(source_dir))
if path_exists == True:
  file_list = fs.listFiles(spark._jvm.org.apache.hadoop.fs.Path(source_dir), True)
  while file_list.hasNext():
    file = file_list.next()
    list_of_files.append(str(file.getPath()))

for file in list_of_files:
  dbutils.fs.mv(file, os.path.join(dest_dir, os.path.basename(file)), recurse=True)
user_98857860
Related