When running Spark jobs on top of YARN (yarn-cluster mode), YARN creates the workers in containers that have a name that looks something like this: container_e116_1495951495692_11203_01_000105
What is the naming convention for the containers?
Here is my educated guess:
- container - Just a constant string, obviously
- e116 - No Idea what this is. Maybe something to do with the YARN version.
- 1495951495692_11203 - The application-id
- 01 - An attempt counter?
- 000105 - This is probably just an increment integer.
If there is any concrete information about this (or even a refference to the right place in the code), I'd be glad to hear about it.
In light of the above, when running a Spark job on YARN, How can I know which containers belong to which executor?