Azure ML Studio notebook using Python 3.6 kernel instead of 3.8 kernel for notebook magic %%bash commands?

user_16708670 0
Viewed 498

I'm working on a Tensorflow project in Azure ML Studio right now and I'm currently following along with this Colab Notebook to learn how to use multiple workers. Whenever I try to run magic %%bash commands in the notebook it seems like the notebook is using the AzureML Python 3.6 kernel instead of the AzureML Python 3.8 Kernel.

The 3.8 kernel is the kernel where I have installed all my necessary python packages and it is the only kernel running in my ML Studio compute. I have confirmed this by checking the kernel manager and making sure that no other kernels are running.

When I try to run

%%bash --bg
python main.py &> job_0.log

I get this error in my job_0.log file...

2021-09-29 23:00:35.170807: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2021-09-29 23:00:35.170899: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2021-09-29 23:00:35.170908: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Traceback (most recent call last):
  File "main.py", line 12, in <module>
    strategy = tf.distribute.MultiWorkerMirroredStrategy()
AttributeError: module 'tensorflow_core._api.v2.distribute' has no attribute 'MultiWorkerMirroredStrategy'

MultiWorkerMirroredStrategy was added in Tensorflow 2.6 so I made sure that I had Tensorflow 2.6 installed in my Python 3.8 kernel. When I pip show in terminal it shows that it is indeed installed there. However when I

%%bash
pip show tensorflow

in my notebook, the output displays this.

Name: tensorflow
Version: 2.1.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /anaconda/envs/azureml_py36/lib/python3.6/site-packages
Requires: astor, six, tensorflow-estimator, protobuf, scipy, keras-applications, opt-einsum, google-pasta, wheel, termcolor, keras-preprocessing, tensorboard, absl-py, numpy, gast, grpcio, wrapt
Required-by: autokeras

I don't know why it is showing the AzureML Python 3.6 installation location when I am running the AzureML Python 3.8 kernel. For context whenever I start a notebook in ML studio the default kernel is AzureML Python 3.6 and I change to Python 3.8 before I begin to run my code.

Also I'm not sure if this is related but when I pip install packages in this ML Studio Compute the default installation goes to the AzureML 3.8 kernel. So I am especially confused. If anyone has experience with this or has had a similar problem I would really appreciate some guidance.

azure-machine-learning-service tensorflow kernel bash
1 Answers

Okay I figured it out. AzureML Studio Kernels are just different conda environments. If you run

conda env list

you'll see all the available default kernel options as environments. The default kernel when running magic %%bash commands is the AzureML Python 3.6 kernel. To change it you just need to run

source activate env_name

Also, activating a new environment will only activate it for that %%bash cell. If you want to use a non-default kernel across multiple %%bash cells then you need to activate the environment at the start of every cell.

If anyone knows how to permanently change this so that the environment does not have to be activated multiple times for every %%bash cell please comment below.

user_167086700
Related