How to get list of all leaf folders from ADLS Gen2 path via Scala code?

user_11704105 0
Viewed 313

We have folders and subfolders in it with year,month, day folders in it. How can we get only the last leaf level folder list using dbutils.fs.ls utility?

Example path:

abfss://cont@storage.dfs.core.windows.net/customer/data/V1/2021/ abfss://cont@storage.dfs.core.windows.net/customer/data/V1/2022/ abfss://cont@storage.dfs.core.windows.net/customer/data/V1/2022/03/24/15/a.parquet abfss://cont@storage.dfs.core.windows.net/customer/data/V1/2022/03/25/15/b.parquet . .

The function should return only last leaf level folder list i.e

abfss://cont@storage.dfs.core.windows.net/customer/data/V1/2022/03/24/15 abfss://cont@storage.dfs.core.windows.net/customer/data/V1/2022/03/25/15

EDIT:

I have tried below function and it works but it fails when some folder is empty with error "java.lang.UnsupportedOperationException: empty.reduceLeft". Please help.

def listLeafDirectories(path: String): Array[String] =
  dbutils.fs.ls(path).map(file => {
    // Work around double encoding bug
    val path = file.path.replace("%25", "%").replace("%25", "%")
    if (file.isDir) listLeafDirectories(path)
    else Array[String](path.substring(0,path.lastIndexOf("/")+1))
  }).reduce(_ ++ _).distinct
azure-databricks databricks apache-spark scala
2 Answers

The function should return only last leaf level folder list

You can use fold function of scala.

def fold[A1 >: A](z: A1)(op: (A1, A1) => A1): A1

Folds the elements of this list using the specified associative binary operator. The default implementation in IterableOnce is equivalent to foldLeft but may be overridden for more efficient traversal orders.

The order in which operations are performed on elements is unspecified and may be nondeterministic.

returns the result of applying the fold operator op between all the elements and z, or z if this list is empty.

For more information refer this link

user_111048050

Below function worked for me

 def listDirectories(dir: String, recurse: Boolean): Array[String] = {
    dbutils.fs.ls(dir).map(file => {
      val path = file.path.replace("%25", "%").replace("%25", "%")
      if (file.isDir) listDirectories(path,recurse)
      else Array[String](path.substring(0, path.lastIndexOf("/")+1))
    }).reduceOption(_ union _).getOrElse(Array()).distinct
  }
user_117041050
Related