FileUtil.copyMerge() in AWS S3

user_6435637 0
Viewed 1972

I have Loaded a DataFrame into HDFS as text format using below code. finalDataFrame is the DataFrame

finalDataFrame.repartition(1).rdd.saveAsTextFile(targetFile)

After executing the above code I found that a directory created with the file name I provided and under the directory a file created but not in text format. The file name is like part-00000.

I have resolved this in HDFS using below code.

val hadoopConfig = new Configuration()
val hdfs = FileSystem.get(hadoopConfig)
FileUtil.copyMerge(hdfs, new Path(srcPath), hdfs, new Path(dstPath), true, hadoopConfig, null)

Now I can get the text file in the mentioned path with given file name.

But when I am trying to do the same operation in S3 it is showing some exception

FileUtil.copyMerge(hdfs, new Path(srcPath), hdfs, new Path(dstPath), true, hadoopConfig, null)

java.lang.IllegalArgumentException: Wrong FS:
s3a://globalhadoop/data, expected:
hdfs://*********.aws.*****.com:8050

It seems that S3 path is not supporting over here. Can anyone please assist how to do this.

hdfs apache-spark hadoop amazon-s3 scala
2 Answers

I have solved the problem using below code.

def createOutputTextFile(srcPath: String, dstPath: String, s3BucketPath: String): Unit = {
    var fileSystem: FileSystem = null
    var conf: Configuration = null
    if (srcPath.toLowerCase().contains("s3a") || srcPath.toLowerCase().contains("s3n")) {
      conf = sc.hadoopConfiguration
      fileSystem = FileSystem.get(new URI(s3BucketPath), conf)
    } else {
      conf = new Configuration()
      fileSystem = FileSystem.get(conf)
    }
    FileUtil.copyMerge(fileSystem, new Path(srcPath), fileSystem, new Path(dstPath), true, conf, null)
  }

I have written the code for filesystem of S3 and HDFS and both are working fine.

user_64356370

You are passing in the hdfs filesystem as the destination FS in FileUtil.copyMerge. You need to get the real FS of the destination, which you can do by calling Path.getFileSystem(Configuration) on the destination path you have created.

user_22612740
Related