Java Copying File in HDFS to another Directory in HDFS

user_2287816 0
Viewed 3718

I'm using the example in this link here to copy contents from one directory in hdfs to another directory in hdfs. The copying of file works, but it creates a new subdirectory in the target vs. just copying the file to the target dir. Example:

  Path source=new Path("hdfs://HANameService/sources/hpm_support/apc_code/");
  Path target=new Path("hdfs://HANameService/staging/hpm_support/apc_code/");
  FileSystem fs = source.getFileSystem(conf); 
  FileUtil.copy(fs, source, fs, target, true, conf);`

So instead of copying the file to hdfs://HANameService/staging/hpm_support/apc_code it creates a new dir under apc_code and the file ends up in hdfs://HANameService/staging/hpm_support/apc_code/apc_code How can I get it to not create that sub-directory?

hdfs hadoop java
2 Answers

As already mentioned by @ernest_k this behavior only occurs if the target directory already exists in HDFS.

So, deleting the target directory before calling the copy method will also solve your problem:

if(fileSystem.exists(pathTo)) {
    fileSystem.delete(pathTo, true);
}

FileUtil.copy(fileSystem, pathFrom, fileSystem, pathTo, false, true, conf);
user_48462150
Related