AWS Glue 3.0 container not working for Jupyter notebook local development

user_3268269 0
Viewed 1399

I am working on Glue in AWS and trying to test and debug in local dev. I follow the instruction here https://aws.amazon.com/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/ to develop Glue job locally. On that post, they use Glue 1.0 image for testing and it works as it should be. However when I load and try to dev by Glue 3.0 version; I follow the guidance steps but, I can't open Jupyter notebook on :8888 like the post said even every step seems correct.

here my cmd to start a Jupyter notebook on Glue 3.0 container

docker run -itd -p 8888:8888 -p 4040:4040 -v ~/.aws:/root/.aws:ro --name glue3_jupyter amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/jupyter/jupyter_start.sh

nothing shows on http://localhost:8888.

still have no idea why! I understand the diff. between versions of Glues just wanna develop and test on the latest version of it. Have anybody got the same issue? Thanks.

aws-glue jupyter-notebook amazon-web-services
1 Answers

It seems that GLUE 3.0 image has some issues with SSL. A workaround for working locally is to disable SSL (you also have to change the script paths as documentation is not updated).

$ docker run -it -p 8888:8888 -p 4040:4040 -e DISABLE_SSL="true" \
    -e AWS_ACCESS_KEY_ID=$(aws --profile default configure get aws_access_key_id) \ 
    -e AWS_SECRET_ACCESS_KEY=$(aws --profile default configure get aws_secret_access_key) \ 
    -e AWS_DEFAULT_REGION=$(aws --profile default configure get region) \
    --name glue_jupyter amazon/aws-glue-libs:glue_libs_3.0.0_image_01 \ 
    /home/glue_user/jupyter/jupyter_start.sh

After a few seconds you should have a working jupyter notebook instance running on http://127.0.0.1:8888

user_15737120
Related