Send a file from Hadoop to a remote server via sftp

user_3617165 0
Viewed 141

I have some files in a directory on HDFS and I need to move them via SFTP to a remote server. Normally, what I do is that I download the files from the HDFS to a unix folder with

hdfs dfs -get /hdfs_path/folder/file.txt /unix_path/dest/path

and then i move it with sftp as:

echo "put /unix_path/dest/path/file.txt /remote_folder" | sftp -b - "user@$remoteServer"

What I want to know: is there any way to do the direct sending of the file via sftp to the remote server from hadoop without needing to do the previous hdfs dfs -get?

hdfs sftp hadoop unix linux
1 Answers

Nifi was created to specifically handle this type of file movement. (It is a separate install). You should check it out. I am not aware of another way of doing it unless you wrote some code in spark.

If you want a smaller footprint I would suggest using Minifi a sub-project of Nifi. Add getHDFS to Minifi. This can manage moving data and could be launched from commandline (but Also could run as an agent sending data)

user_135351200
Related