All notebooks on the Google Dataproc return the root, /, when running !pwd or os.getcwd(). They are actually located in deeper locations on the filesystem, ex: /jupyter/users. This problem seems to be specific to the Google Dataproc. Any ideas to get the location?
I've tried changing the command that starts the jupyter server but then the notebooks just think that path is where they are located.
/opt/conda/miniconda3/bin/jupyter notebook /root/ &> /var/log/jupyter_notebook.log
Now all files think they are located in /root/
