Using a custom docker with Azure ML

Have no issues installing the lib. First, please dump your dockerfile content into a file, easier to maintain and read ;)

e = Environment("custom")
e.docker.base_dockerfile = "path/to/your/dockerfile"

will set the content of the file into a string prop.

e.register(ws).build(ws).wait_for_completion()

step 2/16 will have your apt update and libgl1 install

Note, that should work with >=1.7 SDK

This should work :

from azureml.core import Workspace
from azureml.core.environment import Environment
from azureml.train.estimator import Estimator
from azureml.core.conda_dependencies import CondaDependencies
from azureml.core import Experiment

ws = Workspace (...)
exp = Experiment(ws, 'test-so-exp')

myenv = Environment(name = "myenv")
myenv.docker.enabled = True
dockerfile = r"""
FROM mcr.microsoft.com/azureml/base:intelmpi2018.3-ubuntu16.04
RUN apt-get update && apt-get install -y libgl1-mesa-glx
RUN echo "Hello from custom container!"
"""
myenv.docker.base_image = None
myenv.docker.base_dockerfile = dockerfile

## You need to instead put your packages in the Environment definition instead... 
## see below for some changes too

myenv.python.conda_dependencies = CondaDependencies.create(pip_packages = ['scipy==1.1.0', 'torch==1.5.1'])

Finally you can build your estimator a bit differently :

est = Estimator(
    source_directory = '.',
#     script_params = script_params,
#     use_gpu = True,
    compute_target = 'gpu-cluster-1',
#     pip_packages = ['scipy==1.1.0', 'torch==1.5.1'],
    entry_script = 'AzureEntry.py',
    environment_definition=myenv
    )

And submit it :

run = exp.submit(config = est)
run.wait_for_completion(show_output=True)

Let us know if that works.

Totally understandable why you're struggling -- others have also expressed a need for more information.

1. perhaps base_dockerfile needs to be a text file (with the contents inside) and not a string? I'll ask the environments PM to learn more specifically how this works
2. another option would be to lever Azure Container Instance (ACI). An ACI is created automatically when spinning up an Azure ML workspace. See this GitHub issue for more info on that.

For more information about using Docker in environments, see the article `Enable Docker https://docs.microsoft.com/azure/machine-learning/how-to-use-environments#enable-docker
The following example shows how to load docker steps as a string.

   from azureml.core import Environment
   myenv = Environment(name="myenv")

   # Creates the environment inside a Docker container.
   myenv.docker.enabled = True

   # Specify docker steps as a string.
   dockerfile = r'''
   FROM mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04
   RUN echo "Hello from custom container!"
   '''

   # Alternatively, load from a file.
   #with open("dockerfiles/Dockerfile", "r") as f:
   #    dockerfile=f.read()

   myenv.docker.base_dockerfile = dockerfile

I think it's that you're using an estimator. Estimators create their own environment, unless you set the environment_definition parameter, which I don't see in your snippet. I'm looking at https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.estimator.estimator?view=azure-ml-py.

Haven't tried it, but I think you can fix this by changing your code to:

est = Estimator(
    source_directory = '.',
    script_params = script_params,
    use_gpu = True,
    compute_target = 'gpu-cluster-1',
    pip_packages = ['scipy==1.1.0', 'torch==1.5.1'],
    entry_script = 'AzureEntry.py',
    environment_definition = myenv
    )

run = exp.submit(config = est)
run.wait_for_completion(show_output=True)

You might also have to move use_gpu setting into the environment definition, as the SDK page I linked above says that the environment will take precedence over this and a couple other estimator parameters.