I am currently trying to setup a JupyterLab server in a Docker container which will need to connect to a YARN cluster. The primary use-case is to allow users to kick off Spark jobs interactively in a Jupyter notebook environment. So far I have setup my kernels and corresponding run.sh startup scripts to point to the correct cluster configuration. My success goes as far as communication w/ the Jupyter kernel and the YARN cluster, but whenever I startup a cluster it seems like it would just hang in ACCEPTED state until it eventually time out/used all its retries.
I am considering trying out Jupyter Enterprise Gateway, but that seems like a lot of additional work that would probably require additional infrastructure setup that we currently do not have in place.
Has anyone had any luck w/ getting dockerized Jupyter notebooks to play nice w/ external YARN clusters? If so, how were you able to get it set up?