We are running an ETL job using Spark 3.0.1. We are using an AWS EMR cluster and running the executors using a yarn cluster. What is currently happening is that when the job starts we see the "Container Pending" (from cloudwatch graph) value to be ~100K where as the final number of "Containers Allocated" is much less (~1K) which is what we expect. We are submitting these jobs to Spark using an Airflow LivyOperator where we specify the number of executors accurately. Any idea why "Container Pending" is so high?
Thanks