check if folder exists in blob storage and add to a list

user_6432418 0
Viewed 587

I am creating a function in Databricks to check if folder exists in Azure Blob Storage:

import java.io.File

def checkFolder(paths: List[String]): Unit = {
  
  for (f <- paths) {
    try
      {   
        var pathCheck = dbutils.fs.ls(f) 
       }
  catch {
    case ex: Exception => {
      println("Folder does not exists: " + f)
      }
    }
  }
}

I want to add the folders that exists in list so that I can read the data in those paths! I also want to add the path that do not exist in a different list.

How would I do that?

azure-blob-storage databricks apache-spark scala
1 Answers

Try something like this:

val (exist, doesnt) = paths.map {
  f => try {
    dbutils.fs.ls(f)
    (f, true)
  } catch {
    case _: Exception => 
      (f, false)
  }
}.partition(_._2)

This will give you two sequences of type (String, Boolean) for existing & not existing files, then you can extract file paths from it using:

exist.map(_._1)
doesnt.map(_._1)
Alex_Ott_186270
Related