Working on a project to write parquet files to a object store from k8 spark operator using multiple partition keys. The output location of the spark jobs for the parquet files is the same and the data is written to the subfolders based on the partition keys. What are the options to write the parquet files without HDFS? The default S3A (algo1 and algo2) committers will not work since the temp files in the output location gets deleted after each successful write as data is continuously written to the output location as small batches. is there any way use the directory committer without HDFS? NFS does not seem to work.
0 Answers
Related