I have a scenario, where we have multiple source files(terabytes of data) in csv format, which was tarred and gunzipped as EXAMPLE.tar.gz. I want to process those files in pyspark. Then we are moving it to hdfs. I want to untar EXAMPLE.tar.gz in hdfs before processing in spark.
I tried the below command. But it's not working.
hdfs dfs -cat <hdfs path>/EXAMPLE.tar.gz | tar xvfz | hdfs dfs -put - <new hdfs location>
Any best approach to handle this scenario in linux?