EMR Cluster no visible on AWS Console UI

user_4051090 0
Viewed 1074

I created a Cluster using the code :

> StepFactory stepFactory = new StepFactory();

            StepConfig enableDebugging = new StepConfig().withName("Enable Debugging")
                    .withActionOnFailure("TERMINATE_JOB_FLOW").withHadoopJarStep(stepFactory.newEnableDebuggingStep());


            Application spark = new Application().withName("Spark");

            RunJobFlowRequest createClusterParameters = new RunJobFlowRequest().withName("CreateDatamart")
                                            .withReleaseLabel("emr-5.5.0")
                                            .withSteps(enableDebugging)
                                            .withApplications(spark)
                                            .withLogUri("s3://logs/")
                                            .withServiceRole("EMR_DefaultRole")
                                            .withJobFlowRole("EMR_EC2_DefaultRole")
                                            .withInstances(new JobFlowInstancesConfig()
                                                    .withEc2KeyName("keypair")
                                                    .withInstanceCount(3)
                                                    .withKeepJobFlowAliveWhenNoSteps(false)
                                                    .withMasterInstanceType("m3.xlarge")
                                                    .withSlaveInstanceType("m3.xlarge"));

            RunJobFlowResult createCluster = emr.runJobFlow(createClusterParameters);

The cluster gets created. The steps attached to it are also running. But the cluster is not visible in AWS EMR UI. I can see the details under the Events Tab in EMR. Since it is coming under the Events tab, there is not point that it might be created in some other region. (Although i have checked that too) In the EC2 console i can see the containers created for the EMR.

In the other-case if i create a cluster directly from UI it is visible.

Is there some error with the code?

amazon-emr apache-spark amazon-web-services java
1 Answers

If you are adding IAM user visibility to a new cluster, call RunJobFlow and set VisibleToAllUsers to true, otherwise IAM users cannot view the cluster.

So you'll just need to add the following line before creating your cluster:

createClusterParameters.setVisibleToAllUsers(true);

Note: Is it similar if you are creating an AWS Lambda with the Python SDK:

response1 = emr.run_job_flow(
    Name=CLUSTER_NAME, # more properties [...]
    VisibleToAllUsers=True
)
user_34154090
Related