Hi I am processing binary files with Spark on Java. I know that text files (sc.textFile(path)) should be automatically decompressed if the extension of the file is ".gz".
Is there any similar automatic way to decompress binary data? Should I set some option I may need for it to work? I tried this code and it seems that automatic decompression does not happen.
JavaSparkContext sc = new JavaSparkContext(sparkConf);
String inputFolder = args[0];
JavaPairRDD<String, PortableDataStream> binaryFiles = sc.binaryFiles(inputFolder);