SageMaker Studio PyTorch 1.8 kernel has no PyTorch, Numpy, or Matplotlib module

Viewed 309

I'm working with SageMaker studio with the following options:

  • kernel: PyTorch 1.8 Python 3.6 GPU optimized.
  • instance: ml.g4dn.xlarge

When running import torch numpy, matplotlib or PIL, I'm getting the No module named 'X' error. No matter when using pip install in a cell above, it will not be imported. Is this a problem only I am encountering with the new PyTorch 1.8 kernel? It also happens with the CPU-optimized version. However, PyTorch 1.6 kernel does not throw an error.

When running conda list, I get the following output:

# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             4.5                       1_gnu  
apex                      0.1                      pypi_0    pypi
argon2-cffi               21.1.0                   pypi_0    pypi
argparse                  1.4.0                    pypi_0    pypi
async-generator           1.10                     pypi_0    pypi
attrs                     21.2.0                   pypi_0    pypi
autovizwidget             0.19.1                   pypi_0    pypi
aws-cdk-assets            1.128.0                  pypi_0    pypi
aws-cdk-aws-apigateway    1.128.0                  pypi_0    pypi
aws-cdk-aws-applicationautoscaling 1.128.0                  pypi_0    pypi
aws-cdk-aws-autoscaling   1.128.0                  pypi_0    pypi
aws-cdk-aws-autoscaling-common 1.128.0                  pypi_0    pypi
aws-cdk-aws-autoscaling-hooktargets 1.128.0                  pypi_0    pypi
aws-cdk-aws-batch         1.128.0                  pypi_0    pypi
aws-cdk-aws-certificatemanager 1.128.0                  pypi_0    pypi
aws-cdk-aws-cloudformation 1.128.0                  pypi_0    pypi
aws-cdk-aws-cloudfront    1.128.0                  pypi_0    pypi
aws-cdk-aws-cloudwatch    1.128.0                  pypi_0    pypi
aws-cdk-aws-codebuild     1.128.0                  pypi_0    pypi
aws-cdk-aws-codecommit    1.128.0                  pypi_0    pypi
aws-cdk-aws-codeguruprofiler 1.128.0                  pypi_0    pypi
aws-cdk-aws-codestarnotifications 1.128.0                  pypi_0    pypi
aws-cdk-aws-cognito       1.128.0                  pypi_0    pypi
aws-cdk-aws-dynamodb      1.128.0                  pypi_0    pypi
aws-cdk-aws-ec2           1.128.0                  pypi_0    pypi
aws-cdk-aws-ecr           1.128.0                  pypi_0    pypi
aws-cdk-aws-ecr-assets    1.128.0                  pypi_0    pypi
aws-cdk-aws-ecs           1.128.0                  pypi_0    pypi
aws-cdk-aws-efs           1.128.0                  pypi_0    pypi
aws-cdk-aws-elasticloadbalancing 1.128.0                  pypi_0    pypi
aws-cdk-aws-elasticloadbalancingv2 1.128.0                  pypi_0    pypi
aws-cdk-aws-events        1.128.0                  pypi_0    pypi
aws-cdk-aws-fsx           1.128.0                  pypi_0    pypi
aws-cdk-aws-globalaccelerator 1.128.0                  pypi_0    pypi
aws-cdk-aws-iam           1.128.0                  pypi_0    pypi
aws-cdk-aws-imagebuilder  1.128.0                  pypi_0    pypi
aws-cdk-aws-kinesis       1.128.0                  pypi_0    pypi
aws-cdk-aws-kms           1.128.0                  pypi_0    pypi
aws-cdk-aws-lambda        1.128.0                  pypi_0    pypi
aws-cdk-aws-logs          1.128.0                  pypi_0    pypi
aws-cdk-aws-route53       1.128.0                  pypi_0    pypi
aws-cdk-aws-route53-targets 1.128.0                  pypi_0    pypi
aws-cdk-aws-s3            1.128.0                  pypi_0    pypi
aws-cdk-aws-s3-assets     1.128.0                  pypi_0    pypi
aws-cdk-aws-sam           1.128.0                  pypi_0    pypi
aws-cdk-aws-secretsmanager 1.128.0                  pypi_0    pypi
aws-cdk-aws-servicediscovery 1.128.0                  pypi_0    pypi
aws-cdk-aws-signer        1.128.0                  pypi_0    pypi
aws-cdk-aws-sns           1.128.0                  pypi_0    pypi
aws-cdk-aws-sns-subscriptions 1.128.0                  pypi_0    pypi
aws-cdk-aws-sqs           1.128.0                  pypi_0    pypi
aws-cdk-aws-ssm           1.128.0                  pypi_0    pypi
aws-cdk-cloud-assembly-schema 1.128.0                  pypi_0    pypi
aws-cdk-core              1.128.0                  pypi_0    pypi
aws-cdk-custom-resources  1.128.0                  pypi_0    pypi
aws-cdk-cx-api            1.128.0                  pypi_0    pypi
aws-cdk-region-info       1.128.0                  pypi_0    pypi
aws-parallelcluster       3.0.0                    pypi_0    pypi
awscli                    1.20.63                  pypi_0    pypi
awsio                     0.0.1                    pypi_0    pypi
backcall                  0.2.0                      py_0    anaconda
bcrypt                    3.2.0                    pypi_0    pypi
beautifulsoup4            4.10.0                   pypi_0    pypi
blas                      1.0                    openblas    anaconda
bleach                    4.1.0                    pypi_0    pypi
blis                      0.7.4                    pypi_0    pypi
bokeh                     2.3.3                    pypi_0    pypi
boto3                     1.18.63                  pypi_0    pypi
botocore                  1.21.63                  pypi_0    pypi
bottleneck                1.3.2                    pypi_0    pypi
brotlipy                  0.7.0           py36h8f6f2f9_1001    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2021.9.30            h06a4308_1  
cairo                     1.16.0            h18b612c_1001    conda-forge
catalogue                 2.0.6                    pypi_0    pypi
cattrs                    1.0.0                    pypi_0    pypi
certifi                   2021.5.30        py36h06a4308_0  
cffi                      1.14.6           py36hc120d54_0    conda-forge
chardet                   4.0.0            py36h5fab9bb_1    conda-forge
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.0.3                    pypi_0    pypi
clickclick                20.10.2                  pypi_0    pypi
cloudpickle               2.0.0                    pypi_0    pypi
cmake                     3.18.2.post1             pypi_0    pypi
colorama                  0.4.3                    pypi_0    pypi
conda                     4.10.3           py36h06a4308_0  
conda-package-handling    1.7.3            py36h8f6f2f9_0    conda-forge
connexion                 2.7.0                    pypi_0    pypi
constructs                3.3.161                  pypi_0    pypi
contextvars               2.4                      pypi_0    pypi
cryptography              35.0.0           py36hb60f036_0    conda-forge
cycler                    0.10.0                   pypi_0    pypi
cymem                     2.0.5                    pypi_0    pypi
cython                    0.29.21          py36he6710b0_0    anaconda
dataclasses               0.8                      pypi_0    pypi
decorator                 4.4.2                      py_0    anaconda
defusedxml                0.7.1                    pypi_0    pypi
dgl-cuda11.1              0.6.1                    py36_0    dglteam
dill                      0.3.4                    pypi_0    pypi
docutils                  0.15.2                   pypi_0    pypi
entrypoints               0.3                      pypi_0    pypi
fastai                    1.0.61                   pypi_0    pypi
fastprogress              1.0.0                    pypi_0    pypi
ffmpeg                    4.0                  hcdf2ecd_0  
filelock                  3.3.1                    pypi_0    pypi
flask                     2.0.2                    pypi_0    pypi
fontconfig                2.13.1            hba837de_1005    conda-forge
freeglut                  3.2.1                h9c3ff4c_2    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
fsspec                    2021.10.1                pypi_0    pypi
future                    0.18.2                   py36_1    anaconda
gevent                    21.8.0                   pypi_0    pypi
glib                      2.69.1               h5202010_0  
google-pasta              0.2.0                    pypi_0    pypi
graphite2                 1.3.13            h58526e2_1001    conda-forge
greenlet                  1.1.2                    pypi_0    pypi
h5py                      2.8.0            py36h989c5e5_3  
harfbuzz                  1.8.8                hffaf4a1_0  
hdf5                      1.10.2               hc401514_3    conda-forge
hdijupyterutils           0.19.1                   pypi_0    pypi
horovod                   0.21.3                   pypi_0    pypi
icu                       58.2              hf484d3e_1000    conda-forge
idna                      2.10               pyhd3eb1b0_0  
imageio                   2.9.0                    pypi_0    pypi
immutables                0.16                     pypi_0    pypi
importlib-metadata        4.8.1                    pypi_0    pypi
importlib-resources       5.2.2                    pypi_0    pypi
inflection                0.5.1                    pypi_0    pypi
inotify-simple            1.2.1                    pypi_0    pypi
intel-openmp              2020.2                      254    anaconda
ipykernel                 5.5.6                    pypi_0    pypi
ipython                   7.16.1           py36h5ca1d4c_0    anaconda
ipython_genutils          0.2.0                    py36_0    anaconda
ipywidgets                7.6.5                    pypi_0    pypi
isodate                   0.6.0                    pypi_0    pypi
itsdangerous              2.0.1                    pypi_0    pypi
jasper                    2.0.14               hd8c5072_2  
jedi                      0.18.0           py36h06a4308_1  
jinja2                    3.0.2                    pypi_0    pypi
jmespath                  0.10.0                   pypi_0    pypi
joblib                    1.0.1              pyhd3eb1b0_0  
jpeg                      9d                   h36c2ea0_0    conda-forge
jsii                      1.39.0                   pypi_0    pypi
jsonpatch                 1.32                     pypi_0    pypi
jsonpointer               2.1                      pypi_0    pypi
jsonschema                3.2.0                    pypi_0    pypi
jupyter                   1.0.0                    pypi_0    pypi
jupyter-client            7.0.6                    pypi_0    pypi
jupyter-console           6.4.0                    pypi_0    pypi
jupyter-core              4.8.1                    pypi_0    pypi
jupyterlab-pygments       0.1.2                    pypi_0    pypi
jupyterlab-widgets        1.0.2                    pypi_0    pypi
kiwisolver                1.3.1                    pypi_0    pypi
ld_impl_linux-64          2.35.1               h7274673_9  
libffi                    3.3                  he6710b0_2  
libgcc                    7.2.0                h69d50b8_2  
libgcc-ng                 9.3.0               h5101ec6_17  
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0    anaconda
libglu                    9.0.0             he1b5a44_1001    conda-forge
libgomp                   9.3.0               h5101ec6_17  
libopenblas               0.3.10               h5a2b251_0    anaconda
libopencv                 3.4.2                hb342d67_1  
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libstdcxx-ng              9.3.0               hd4cf53a_17  
libtiff                   4.0.10            hc3755c2_1005    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libvpx                    1.7.0                h439df22_0  
libxcb                    1.13              h7f98852_1003    conda-forge
libxml2                   2.9.12               h03d6c58_0  
llvmlite                  0.36.0                   pypi_0    pypi
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
magma-cuda111             2.5.2                         1    pytorch
markupsafe                2.0.1                    pypi_0    pypi
marshmallow               3.13.0                   pypi_0    pypi
matplotlib                3.3.4                    pypi_0    pypi
mistune                   0.8.4                    pypi_0    pypi
mkl                       2020.2                      256    anaconda
mkl-include               2020.2                      256    anaconda
mock                      4.0.3                    pypi_0    pypi
mpi4py                    3.0.3                    pypi_0    pypi
multiprocess              0.70.12.2                pypi_0    pypi
murmurhash                1.0.5                    pypi_0    pypi
nbclient                  0.5.4                    pypi_0    pypi
nbconvert                 6.0.7                    pypi_0    pypi
nbformat                  5.1.3                    pypi_0    pypi
ncurses                   6.2                  he6710b0_1  
nest-asyncio              1.5.1                    pypi_0    pypi
networkx                  2.5.1              pyhd3eb1b0_0  
nose                      1.3.7                    pypi_0    pypi
notebook                  6.4.4                    pypi_0    pypi
numba                     0.53.1                   pypi_0    pypi
numexpr                   2.7.3                    pypi_0    pypi
numpy                     1.19.1           py36h30dfecb_0    anaconda
numpy-base                1.19.1           py36h75fe3a5_0    anaconda
nvidia-ml-py3             7.352.0                  pypi_0    pypi
openapi-schema-validator  0.1.5                    pypi_0    pypi
openapi-spec-validator    0.3.1                    pypi_0    pypi
opencv                    3.4.2            py36h6fd60c2_1  
opencv-python             4.5.3.56                 pypi_0    pypi
openssl                   1.1.1l               h7f8727e_0  
packaging                 21.0                     pypi_0    pypi
pandas                    1.1.5            py36ha9443f7_0  
pandocfilters             1.5.0                    pypi_0    pypi
paramiko                  2.8.0                    pypi_0    pypi
parso                     0.8.0                      py_0    anaconda
pathos                    0.2.8                    pypi_0    pypi
pathy                     0.6.0                    pypi_0    pypi
pcre                      8.45                 h9c3ff4c_0    conda-forge
pexpect                   4.8.0                    py36_0    anaconda
pickleshare               0.7.5                    py36_0    anaconda
pillow                    8.3.2                    pypi_0    pypi
pip                       21.3               pyhd8ed1ab_0    conda-forge
pixman                    0.38.0            h516909a_1003    conda-forge
plotly                    5.3.1                    pypi_0    pypi
pox                       0.3.0                    pypi_0    pypi
ppft                      1.6.6.4                  pypi_0    pypi
preshed                   3.0.5                    pypi_0    pypi
progress                  1.6                      pypi_0    pypi
prometheus-client         0.11.0                   pypi_0    pypi
prompt-toolkit            3.0.8                      py_0    anaconda
protobuf                  3.18.1                   pypi_0    pypi
protobuf3-to-dict         0.1.5                    pypi_0    pypi
psutil                    5.8.0                    pypi_0    pypi
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.6.0                    py36_0    anaconda
publication               0.0.3                    pypi_0    pypi
pure-sasl                 0.6.2                    pypi_0    pypi
py-opencv                 3.4.2            py36hb342d67_1  
pyarrow                   5.0.0                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pybind11                  2.8.0                    pypi_0    pypi
pycosat                   0.6.3           py36h8f6f2f9_1006    conda-forge
pycparser                 2.20                       py_2  
pydantic                  1.8.2                    pypi_0    pypi
pyfunctional              1.4.3                    pypi_0    pypi
pygments                  2.7.1                      py_0    anaconda
pyhive                    0.6.4                    pypi_0    pypi
pyinstrument              3.4.2                    pypi_0    pypi
pyinstrument-cext         0.2.4                    pypi_0    pypi
pykerberos                1.2.1                    pypi_0    pypi
pynacl                    1.4.0                    pypi_0    pypi
pyopenssl                 19.1.0                     py_1    anaconda
pyparsing                 2.4.7                    pypi_0    pypi
pyrsistent                0.18.0                   pypi_0    pypi
pysocks                   1.7.1            py36h5fab9bb_3    conda-forge
python                    3.6.13          hffdb5ce_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd3eb1b0_0  
python_abi                3.6                     2_cp36m    conda-forge
pytz                      2021.3             pyhd3eb1b0_0  
pyyaml                    5.4.1                    pypi_0    pypi
pyzmq                     22.3.0                   pypi_0    pypi
qtconsole                 5.1.1                    pypi_0    pypi
qtpy                      1.11.2                   pypi_0    pypi
readline                  8.1                  h27cfd23_0  
requests                  2.26.0             pyhd3eb1b0_0  
requests-kerberos         0.12.0                   pypi_0    pypi
retrying                  1.3.3                    pypi_0    pypi
rsa                       4.7.2                    pypi_0    pypi
ruamel_yaml               0.15.100         py36h27cfd23_0  
s3fs                      0.4.2                    pypi_0    pypi
s3transfer                0.5.0                    pypi_0    pypi
sagemaker                 2.63.1                   pypi_0    pypi
sagemaker-experiments     0.1.35                   pypi_0    pypi
sagemaker-pytorch-training 2.4.0                    pypi_0    pypi
sagemaker-studio-analytics-extension 0.0.2                    pypi_0    pypi
sagemaker-studio-sparkmagic-lib 0.1.3                    pypi_0    pypi
sagemaker-training        3.9.2                    pypi_0    pypi
sasl                      0.3.1                    pypi_0    pypi
scikit-learn              0.24.2           py36ha9443f7_0  
scipy                     1.5.4                    pypi_0    pypi
seaborn                   0.11.2                   pypi_0    pypi
send2trash                1.8.0                    pypi_0    pypi
setuptools                49.6.0           py36h5fab9bb_3    conda-forge
shap                      0.39.0                   pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_0  
sklearn                   0.0                      pypi_0    pypi
slicer                    0.0.7                    pypi_0    pypi
smart-open                5.2.1                    pypi_0    pypi
smclarify                 0.2                      pypi_0    pypi
smdebug                   1.0.9                    pypi_0    pypi
smdebug-rulesconfig       1.0.1                    pypi_0    pypi
smdistributed-dataparallel 1.2.0                    pypi_0    pypi
smdistributed-modelparallel 1.3.1                    pypi_0    pypi
soupsieve                 2.2.1                    pypi_0    pypi
spacy                     3.1.3                    pypi_0    pypi
spacy-legacy              3.0.8                    pypi_0    pypi
sparkmagic                0.19.1                   pypi_0    pypi
sqlite                    3.36.0               hc218d9a_0  
srsly                     2.4.1                    pypi_0    pypi
tabulate                  0.8.9                    pypi_0    pypi
tenacity                  8.0.1                    pypi_0    pypi
terminado                 0.12.1                   pypi_0    pypi
testpath                  0.5.0                    pypi_0    pypi
thinc                     8.0.10                   pypi_0    pypi
threadpoolctl             2.2.0              pyh0d69192_0  
thrift                    0.15.0                   pypi_0    pypi
thrift-sasl               0.4.3                    pypi_0    pypi
tk                        8.6.10               hbc83047_0  
torch                     1.8.1                    pypi_0    pypi
torchfile                 0.1.0                    pypi_0    pypi
torchnet                  0.0.4                    pypi_0    pypi
torchvision               0.9.1                    pypi_0    pypi
tornado                   6.1                      pypi_0    pypi
tqdm                      4.61.2             pyhd3eb1b0_1  
traitlets                 4.3.3                    py36_0    anaconda
typer                     0.4.0                    pypi_0    pypi
typing                    3.7.4.3                  py36_0    anaconda
typing-extensions         3.10.0.2                 pypi_0    pypi
tzdata                    2021a                h52ac0ba_0  
urllib3                   1.26.6             pyhd3eb1b0_1  
visdom                    0.1.8.9                  pypi_0    pypi
wasabi                    0.8.2                    pypi_0    pypi
wcwidth                   0.2.5                      py_0    anaconda
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.2.1                    pypi_0    pypi
werkzeug                  2.0.2                    pypi_0    pypi
wheel                     0.36.2             pyhd3eb1b0_0  
widgetsnbextension        3.5.1                    pypi_0    pypi
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h7b6447c_0  
yaml                      0.2.5                h7b6447c_0  
zipp                      3.6.0                    pypi_0    pypi
zlib                      1.2.11               h7b6447c_3  
zope-event                4.5.0                    pypi_0    pypi
zope-interface            5.4.0                    pypi_0    pypi
zstd                      1.4.9                ha95c52a_0    conda-forge
1 Answers

I have followed your explanation to recreate the No module named 'X'. After instance lunch, I could import numpy, matplotlib and PIL but could not import torch because torch is not initially installed.

I followed these steps to install and import torch:

  • Shut down your instance completely.
  • Attach the kernel to your notebook again
  • Lunch your ml.g4dn.xlarge instance again
  • pip3 install torch import torch
Related