I'm launching a spark job on my EMR Serverless application, and it hangs for 20 minutes with the same message popping up over and over again :
22/09/11 16:09:39 INFO ExecutorContainerAllocator: Going to request 1 executors for ResourceProfile Id: 0, target: 1 already provisioned: 0.
22/09/11 16:09:39 INFO DefaultEmrServerlessRMClient: Creating containers with container role SPARK_EXECUTOR and keys: Set(249)
22/09/11 16:09:39 INFO DefaultEmrServerlessRMClient: Containers created with container role SPARK_EXECUTOR. key to container id map: Map(249 -> 56c196a8-2492-25fb-617e-1a097bb263da)
22/09/11 16:09:42 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Eventually, the whole thing crashes with the following error :
22/09/11 16:10:07 ERROR EmrServerlessClusterSchedulerBackend: Abandoning job due to no executor being launched within 1200000ms after driver starts.
My maximum capacity is 10vCPU and 80GB. Nothing huge, but the data being manipulated here is tiny (it's just a test run with a CSV with about 10 lines in it). Any clues as to what on earth is happening here ?
UPDATE : Removing my maximum capacity configuration has resolved this. I find this odd as the maximum capacity should have been overwhelmingly sufficient for such a trivial task... As expected with a new service I think cost control and sizing are going to be a little tricky to get the hang of... Still open to any explanations as to why such a trivial task somehow overwhelmed my maximum capacity.