ModuleNotFoundError: Failed to import modules from directory on distributed system

user_18224845 0
Viewed 33

I am working on a distributed system (yarn container), my IDE is Jupyterlab, in addition I have access to a bash terminal. I have several .py scripts in a distinct directory (with init.py), i try to import them as modules in my Notebook.

So far I tried this in bash terminal:

RPATH=$(pwd)
PPATH=$(pwd)'/src' # /src is the module folder
echo $RPATH
echo $PPATH
export PYTHONPATH=$PPATH
export ROOTPATH=$RPATH
echo $PYTHONPATH
echo $ROOTPATH
conda env config vars set PYTHONPATH=$PPATH
conda env config vars set ROOTPATH=$RPATH

And in the notebook itself i tried to set the environment variables by hand:

import os
os.environ['PYTHONPATH']= '/data/yarn/local_dirs/usercache/{user}/appcache/{application_id}/{container_id}/tmp/{working directory}/src'
os.environ['ROOT_DIR']= '/data/yarn/local_dirs/usercache/{user}/appcache/{application_id}/{container_id}/tmp/{working directory}'

This gives me repeatedly:

ModuleNotFoundError: No module named {modulename}

Any ideas are much appreciated!

distributed environment-variables module python
1 Answers

Python reads PYTHONPATH as part of initializing sys.path, but updates to the former will not be reflected in the latter (unless you start a Python subprocess).

user_85862270
Related