List/Retrieve HDFS partitions as Map(String,List(String)) from spark

user_1178028 0
Viewed 778

I wanted to know if there is some way to leverage the metadata information already present in the hdfs folder structure in spark. For example, I am using the following code to write data to hdfs,

val columns = Seq("country","state")
dataframe1.write.partitionBy(columns:_*).mode("overwrite").
save(path)

This generates similar looking directory structure,

path/country=xyz/state=1
path/country=xyz/state=2
path/country=xyz/state=3
path/country=abc/state=4

What i want to know is using spark, is there a way to infer all the partitions and sub partitions as a Map(String,List(String)) (without loading the entire file and using group by?), where the key is the partition and the value is a list of all the sub partitions within this partition.

The output of the above example would be similar to following

Map(xyz->List(1,2,3),abc->(4))
hdfs apache-spark hadoop scala
1 Answers

your hdfs file structure is like this...

$tree path
path
├── country=abc
│   └── state=4
└── country=xyz
    ├── state=1
    ├── state=2
    ├── state=3
    ├── state=4
    ├── state=5
    └── state=6

you need to use this for getting full paths as string..

val lb = new ListBuffer[String]
  def getAllFiles(path:String, sc: SparkContext):Unit = {
  val conf = sc.hadoopConfiguration
    val fs = FileSystem.get(conf)
    val files: RemoteIterator[LocatedFileStatus] = fs.listLocatedStatus(new Path(path))
    while(files.hasNext) {
      var filepath = files.next.getPath.toString
      //println(filepath)
      lb += (filepath)
      getAllFiles(filepath, sc)
    }
    println(lb)
  }

once you get listbuffer with all files full path including subfolders... you need to write the logic to fill in to map. that I am leaving it to you. TIY..

Note : ListBuffer has group by which returns map may be you can use it

in my case I experimented like this...

  println( lb.groupBy(_.toString.replaceAll("file:/Users/xxxxxx/path/country=", "")substring(0, 3) ))

I got the result like 

Map(abc -> ListBuffer(file:/Users/xxxxxx/path/country=abc, file:/Users/xxxxxx/path/country=abc/state=4), xyz -> ListBuffer(file:/Users/xxxxxx/path/country=xyz, file:/Users/xxxxxx/path/country=xyz/state=1, file:/Users/xxxxxx/path/country=xyz/state=6, file:/Users/xxxxxx/path/country=xyz/state=3, file:/Users/xxxxxx/path/country=xyz/state=4, file:/Users/xxxxxx/path/country=xyz/state=5, file:/Users/xxxxxx/path/country=xyz/state=2))

may be you can use this idea to further refine to your desired result.

user_6470530
Related