I am running experiments with some Spark jobs and I am trying to compare performance on EMR and on EKS. The hardware that I used was 2 instances of m5.2xlarge (8 vCore, 32 GiB memory). The reason is that it is commonly shared hardware instances between EKS and EMR and therefore more reliable to compare performance.
I also shared the spark configuration:
--conf spark.executor.instances=2 \
--conf spark.executor.cores=3 \
--conf spark.default.parallelism=16 \
--conf spark.executor.memory=4g \
--conf spark.driver.memory=4g \
--conf spark.executor.memoryOverhead=4g
(Spark 2.4.5 for EMR, Spark 3.0.0 for Kubernetes)
The spark jobs read some json files from S3 and they store parquet on S3 again. I systematically get faster writes and reads from S3 on EMR (23% faster approximately on EMR).
Could that be because of s3-specific optimizations on EMR ? What could be possible things to do to make the performance better on Kubernetes ?