AWS Glue Pyspark Parquet write to S3 taking too long

Question

I have a AWS glue job (PySpark) that needs to load data from a centralized data lake of size 350GB+, prepare it and load into a s3 bucket partitioned by two columns. I noticed that it takes really a long time (around a day even) just to load and write one week of data. There are months of data that needs to be written. I tried increasing the worker nodes but it does not seem to fix the problem.

My glue job currently has 60 G.1x worker nodes.

My SparkConf in the code looks like this

conf = pyspark.SparkConf().setAll([

        ("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", "2"),

        ("spark.speculation", "false"),

        ("spark.sql.parquet.enableVectorizedReader", "false"),

        ("spark.sql.parquet.mergeSchema", "true"),

        ("spark.sql.crossJoin.enabled", "true"),

        ("spark.sql.sources.partitionOverwriteMode","dynamic"),

        ("spark.hadoop.fs.s3.maxRetries", "20"),

        ("spark.hadoop.fs.s3a.multiobjectdelete.enable", "false")

    ])

I believe it does succeed in writing the files into partitions, however it is taking really long to delete all the temporary spark-staging files it created. When I checked the tasks, this seems to take most of the time.

2021-04-22 03:08:50,558 INFO [Thread-6] s3n.S3NativeFileSystem (S3NativeFileSystem.java:rename(1355)): rename s3:///etl/sessions/.spark-staging-8df58afd-d6b2-4ca0-8611-429125abe2ae/p_date=2020-12-16/geo=u1hm s3:///etl/sessions/p_date=2020-12-16/geo=u1hm

My write to S3 looks like this

finalDF.coalesce(50).write.partitionBy('p_date','geohash').save("s3://{bucket}/{basepath}/{table}/".format(bucket=args['DataBucket'], basepath='etl',

                                                         table='sessions'), format='parquet', mode="overwrite")

Any help would be appericiated.

AWS Glue Pyspark Parquet write to S3 taking too long

0 Answers