Hi I am quite new in AWS and emr. I am trying to create a pipeline using emr so I have created a cluster on emr to run a notebook. I am using run_job_flow() to create the cluster and code as below:
import boto3
client = boto3.client('emr', region_name='eu-central-1')
response = client.run_job_flow(
Name="cluster_test02",
ReleaseLabel='emr-6.2.0',
LogUri='s3://aws-logs-002294572453-eu-central-1/elasticmapreduce/',
Applications=[
{
'Name':'hadoop'
},
{
'Name': 'spark',
},
{
'Name': 'livy',
},
{
'Name': 'hive'
},
{
'Name': 'JupyterEnterpriseGateway'
},
],
Instances =
{
'KeepJobFlowAliveWhenNoSteps': True,
'TerminationProtected': False,
'InstanceGroups': [
{
'Name': "Master nodes",
'Market': 'ON_DEMAND',
'InstanceRole': 'MASTER',
'InstanceType': 'm5.xlarge',
'InstanceCount': 1,
},
{
'Name': "Slave nodes",
'Market': 'ON_DEMAND',
'InstanceRole': 'CORE',
'InstanceType': 'm5.xlarge',
'InstanceCount': 1,
},
],
'TerminationProtected': False,
},
Steps=[{
'Name': 'Setup hadoop debugging',
'ActionOnFailure': 'CONTINUE',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': ['state-pusher-script']
}
}],
VisibleToAllUsers=True,
EbsRootVolumeSize=10,
JobFlowRole='EMR_EC2_DefaultRole',
ServiceRole='EMR_DefaultRole',
AutoScalingRole="EMR_AutoScaling_DefaultRole"
)
When I try to attach the cluster to an existing notebook, it doesn't appear on the cluster selection as in the picture attached below. I have also installed the required applications hadoop, spark, livy, hive, JupyterEnterpriseGateway so I am not sure where it actually lacks
