Mapping EMR steps to YARN applications

Question

Am aggregating EMR yarn application logs to S3 using below YARN configuration:

[
  {
    "Classification": "yarn-site",
    "Properties": {
      "yarn.log-aggregation-enable": "true",
      "yarn.log-aggregation.retain-seconds": "-1",
      "yarn.nodemanager.remote-app-log-dir": "s3://mybucket/logs"
    }
  }
]

The logs are grouped by yarn application ID in S3:

s3://mybucket/logs/appplication_id_001/
s3://mybucket/logs/appplication_id_002/
s3://mybucket/logs/appplication_id_003/

I want to map the EMR step ID to YARN application ID, so that given a step ID I will be able to fetch its logs.

The reason why I need this is because am using Apache Airflow for orchestration and would like to fetch the logs and show it it Airflow. My Airflow DAG looks like this:

create-cluster 
    -> add-step-1 -> watch-step-1
    -> add-step-2 -> watch-step-2
    -> add-step-3 -> watch-step-3

At the end of every watch-step-n task I would like to fetch the logs for that step from s3 and print it. Since none of the tasks in the DAG are aware of the YARN application ID am trying to find a way to get the application ID for a step ID.

EDIT

I couldn't find a way to map EMR step ID to YARN application ID. However I was able to group the logs by cluster ID. For example,

s3://mybucket/logs/cluster-id-0/appplication_id_001/
s3://mybucket/logs/cluster-id-0/appplication_id_002/

s3://mybucket/logs/cluster-id-1/appplication_id_001/

I found a way to get cluster ID from within the EMR nodes:

cat /mnt/var/lib/info/job-flow.json | jq -r '.jobFlowID'

I injected the cluster ID (as property named JOB_FLOW_ID) with YARN_OPTS in the yarn-env configuration. And set the yarn.nodemanager.remote-app-log-dir-suffix to JOB_FLOW_ID in yarn-site configuration.

yarn-env configuration:

{
    "Classification": "yarn-env",
    "Properties": {},
    "Configurations": [
        {
            "Classification": "export",
            "Properties": {
                "JOB_FLOW_ID": "\"$(cat /mnt/var/lib/info/job-flow.json | jq -r '.jobFlowId')\"",
                "YARN_OPTS": "\"$YARN_OPTS -Djob_flow_id=$JOB_FLOW_ID\""
            },
            "Configurations": []
        }
    ]
}

yarn-site configuration:

{
    "Classification": "yarn-site",
    "Properties": {
        "yarn.log-aggregation.retain-seconds": "-1",
        "yarn.log-aggregation-enable": "true",
        "yarn.nodemanager.remote-app-log-dir": "s3://mybucket/logs",
        "yarn.nodemanager.remote-app-log-dir-suffix": "${job_flow_id}"
    }
}

Mapping EMR steps to YARN applications

0 Answers