Configuring EMR notebook with github

user_237696 0
Viewed 1615

I have an EMR cluster running and I'm trying to link the notebook to github.

I have:

  • Added AWS secret access to the EMR role (this was my first error)
  • Added outbound HTTPS/443 to all EMR security groups

Attempted to link the notebook to a private repo with my user/pass secret, but I get the following error:

Unable to reach repository https://github.com/<my repo>. Ensure network and security groups have valid configurations. Ensure that the repository information provided is correct.

I see in the docs it talks about NAT and VPG options, both of which I do not have. Is that needed? The docs are really light on details/configuration.

https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-git-considerations.html

Any help is appreciated.

amazon-emr amazon-vpc jupyter-notebook amazon-web-services amazon-ec2
2 Answers

You need to take care of 4 things:

  1. the git repo along with the target branch should exist

  2. configure the credentials to access Git using the Secrets Manager

  3. your EMR cluster mandatorily has to be in private subnet and NOT public subnet. You mandatorily need to have a NAT G/W in your VPC that this private subnet should use to reach out to the internet i.e. configure your route table for this private subnet to map 0.0.0.0/0 to the NAT G/W.

  4. You need to have 2 Security Groups (SG):

    i. A SG for the master instance

    inbound rule - Allow TCP Port 18888 from any resources in the default EC2 security group for EMR Notebooks

    outbound rule - None

    ii. A SG for the notebook instance

    i. A SG for the master instance

    inbound rule - None

    outbound rule - Allow TCP Port 18888 to any resources in the default EC2 security group for EMR Notebooks. Also allow the notebook to route traffic to the internet via the cluster e.g. HTTPS TCP 443 0.0.0.0/0

Source: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-security-groups.html

user_51238420

end-to-end, as follows :

1.) create "private" subnets.
from scratch, I created a dedicated vpc first : say 10.0.16.0/20.
Next, I create an internet gateway. Attach it to the above vpc.

In this VPC, create a "public" subnet first. say 10.0.17.0/24. It is public because of the above internet gateway entry in its route table. This is done by creating a route table under VPC section, and that route table has this internet gateway underneath. This route table is then attached to this subnet 10.0.17.0/24 so now it stands "public".

Now, create another subnet 10.0.20.0/24. Create a NAT Gateway, and make an entry for it in this subnet's route table. This NAT Gateway, must flow into the public subnet created above, i.e 10.0.17.0/24.

Now, you private subnet 10.0.20.0/24 has internet access via NAT gateway interfacing the public subnet above.

2.) Enter "Security Groups". total 4 SGs are at play,
of which, ElasticMapReduceEditors-Editor and ElasticMapReduceEditors-Livy must have outbound to 0.0.0.0/0 allowed. The other two SGs are for Master and Slaves, i.e ElasticMapReduce-Master-Private and ElasticMapReduce-Slave-Private

3.) Make Private Git repositories, Under the settings section of your github profile, create a Personal Access Token. While furnishing details on creating an EMR Repository, add this Secret Value, save it. Your Notebook Service Role must have permission "GetSecretValue" on all the Repositories ie "r-*".
Also, Make sure that the branch specified must have to exist before you configure while creating / linking repositories.

The biggest problem with AWS is that they are great Salesmen but procrastinate when it comes to after-sales / maintaining so many products they make, Their documentations are terrible, lack in end-to-end examples and people have to rely on SO communities. I am sorry friend I am procrastinating too :p I would want to go get this into a Fine Blog and furnish the link in my this answer next!

user_33564240
Related