Reading large gz files in Spark

user_1513514 0
Viewed 874

I have 2 gzip files each around 30GB in size and have written spark code to analyze them. The spark cluster has 4 worker nodes (28GB RAM and 4 cores each) and 2 head nodes ( 64GB RAM).

I am using sc.textFile(histfile,20) to read these 2 gzip files and parallelize them. The job is divided in 3 stages - reduceByKey, reduceByKey and saveAsTextFile. As there are 2 files, only 2 workers are being utilized as of now.

It has taken more than 4 hours and still the 1st stage is not finished. What is the right way to run spark jobs when each gzip file is 100GB-200GB in size.

pyspark apache-spark
1 Answers

gzip file is not spllitable hence your first stage will spawn two map task and it will take a lot of time for the first stage to pass.

Please repartition before the second phase for a good distribution to reduce task.

user_48391570
Related