How to recursively copy all files except empty files including subdirectories in Databricks file system

user3428526 0
Viewed 24

I have a requirement to recursively move all files except empty files (0-byte files) to a destination folder while preserving hierarchies in a Databricks file system. Example:

  folder_path/
   
     1/
        2/
           file1.json- 0 byte
           file2.json- 128 kb
    3/ 
       4/
           file3.json- 0 byte
           file4.json- 20 kb
outputs:
folder_path/
    1/
       2/
          file2.jpg - 128 kb
    3/ 
       4/ file4.jpg - 20 kb

I'm able to implement this using shell script and AWK but it seems Databricks does not support AWK.

Is there any way to implement this either in Python or Scala using Databricks notebook ?

azure-databricks databricks apache-spark-sql
1 Answers

The below implementation works for me. If the size is 0 and does not end with "json", then it is a directory. Else it is a file with 0 size.

import os
src = 'dbfs:/<src_path>'
def createSubDir(dir):
    
    for i in range(0, len(dir)):
            if(dir[i].size == 0 and not dir[i].name.endswith('json')):
                    dbutils.fs.mkdirs(dir[i].path.replace("<src_dir>", "<dest_dir>"))
                    createSubDir(dbutils.fs.ls(dir[i].path))
            elif(dir[i].size != 0 and dir[i].name.endswith('json')):
                    dbutils.fs.mv(dir[i].path, dir[i].path.replace("<src_dir>", "<dest_dir>"))

dir = dbutils.fs.ls(src)
createSubDir(dir)
user34285260
Related