EMR cluster hangs in Step state 'Running/Pending'

Viewed 524

I am launching an EMR cluster through java SDK with a custom jar step. The cluster launch is successful but when after bootstrapping while the step is pending/running state the cluster stucks. I am not even able to ssh on the machine.

Following is my code to launch the cluster with custom jar step -

        String dataTrasnferJar = s3://test/testApplication.jar;
        if (dataTrasnferJar == null || dataTrasnferJar.isEmpty())
            throw new InvalidS3ObjectException(
                    "EMR custom jar file path is null/empty. Please provide a valid jar file path");

        HadoopJarStepConfig customJarConfig = new HadoopJarStepConfig().withJar(dataTrasnferJar);
        StepConfig customJarStep = new StepConfig("Mongo_to_S3_Data_Transfer", customJarConfig)
                .withActionOnFailure(ActionOnFailure.CONTINUE);

        AmazonElasticMapReduce emr = AmazonElasticMapReduceClientBuilder.standard()
                .withCredentials(awsCredentialsProvider)
                .withRegion(region)
                .build();

        Application spark = new Application().withName("Spark");

        String clusterName  = "my-cluster-" + System.currentTimeMillis();
        RunJobFlowRequest request = new RunJobFlowRequest()
                .withName(clusterName)
                .withReleaseLabel("emr-6.0.0")
                .withApplications(spark)
                .withVisibleToAllUsers(true)
                .withSteps(customJarStep)
                .withLogUri(loggingS3Bucket)
                .withServiceRole("EMR_DefaultRole")
                .withJobFlowRole("EMR_EC2_DefaultRole")
                .withInstances(new JobFlowInstancesConfig()
                    .withEc2KeyName(key_pair) 
                    .withInstanceCount(instanceCount)
                    .withEc2SubnetIds(subnetId)
                    .withAdditionalMasterSecurityGroups(securityGroup)
                    .withKeepJobFlowAliveWhenNoSteps(true)    
                    .withMasterInstanceType(instanceType));

        RunJobFlowResult result = emr.runJobFlow(request);  
1 Answers

EMR emr-6.0.0 version is still in development. Can you try the same with emr-5.29.0?

Related