Overwrite to S3 Bucket from EMR taking long time

user_14042443 0
Viewed 555

I am using EMR (5.30) with spark and hadoop as the selected application. The conditions are as followed -

  • Historical data is in S3 location(700 GB)
  • Daily incremental data (500 MB)
  • I need to create a merge framework based on some conditions
  • Write the result back to S3 and terminate the cluster

What I am doing -

  • spinning up the cluster with 1 m5.8xlarge(master) and 5 m5.8xlarge (core) from lambda with all the configurations
  • configurations - 29 executor instances, 5 executor cores, 18 GB memory/ executor. 3 GB overhead memory, default parallelism 290
  • in the PySpark script :
    • copying the historical data after some operations like casting datatype and then to temp location (S3) - overwrite
    • adding incremental data to temp location (same temp location) - append
    • doing all required transformations
    • putting back the result to another S3 location - overwrite, (same bucket but different folder) from where next day same cycle will be continued

Observations:

  • This final overwrite process is taking more than an hour, but first one is taking only 14-15 min
  • I have tried with repartition and coalesce, no improvement
  • When tried with s3n.multipart.uploads.enabled to false, Given error to enable it
  • As per the Amazon documentation https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html , it is having limitation of only 1000 parts in a list. So, per file size is 600-700 MB on an average as the total output size is around 700 GB So, whenever I have tried to override the no. with repartition, it actually could not happen
  • if I use partionBy logic in the code when overwriting, it does not make sense, as it is taking more time(more than 2 hours). Also next day I have to overwrite the result.
  • another observation, when overwriting, It will first delete the contents of the folder, then delete the folder and recreate it to put the result set from EMR. In this case, it failed the whole process in between.

My question is:

  1. Why the both overwrite processes are taking different time intervals where the code is same df.write.mode('overwrite').parquet(target_location)
  2. How can I improve the performance of the last overwrite
amazon-emr aws-glue apache-spark amazon-web-services amazon-s3
1 Answers

IIUC, in your first scenario you are processing 700 GB data and in your second scenario you are performing some join between 500 MB data anad 700 GB data to enable the upserts and saving it back to s3.

If this is the case then, then the problem is not with your write statement instead it is with the transformations you are performing in second case as you must be joining the incremental data with full data to identify the inserts and the updates then modifying the results accordingly.

If this continues your data will keep growing and you will keep running out of time and memory to process your data.

If you are building an incremental datalake over s3 you should consider using HUDI which will fasten up your job and also utilizes your cluster properly for handling just the upserts.

Here is the link to implement the HUDI.

https://github.com/apache/hudi

user_53527480
Related