Using the Environment Class with Pipeline Runs

user_10435556 0
Viewed 751

I am using an estimator step for a pipeline using the Environment class, in order to have a custom Docker image as I need some apt-get packages to be able to install a specific pip package. It appears from the logs that it's completely ignoring, unlike the non-pipeline version of the estimator, the docker portion of the environment variable. Very simply, this seems broken :

I'm running on SDK v1.0.65, and my dockerfile is completely ignored, I'm using 

FROM mcr.microsoft.com/azureml/base:latest\nRUN apt-get update && apt-get -y install freetds-dev freetds-bin vim gcc

in the base_dockerfile property of my code. Here's a snippet of my code : 

from azureml.core import Environment
from azureml.core.environment import CondaDependencies
conda_dep = CondaDependencies()
conda_dep.add_pip_package('pymssql==2.1.1')
myenv = Environment(name="mssqlenv")
myenv.python.conda_dependencies=conda_dep
myenv.docker.enabled = True
myenv.docker.base_dockerfile = 'FROM mcr.microsoft.com/azureml/base:latest\nRUN apt-get update && apt-get -y install freetds-dev freetds-bin vim gcc'
myenv.docker.base_image = None

This works well when I use an Estimator by itself, but if I insert this estimator in a Pipeline, it fails. Here's my code to launch it from a Pipeline run: 

from azureml.pipeline.steps import EstimatorStep

sql_est_step = EstimatorStep(name="sql_step", 
                         estimator=est, 
                         estimator_entry_script_arguments=[],
                         runconfig_pipeline_params=None, 
                         compute_target=cpu_cluster)
from azureml.pipeline.core import Pipeline
from azureml.core import Experiment
pipeline = Pipeline(workspace=ws, steps=[sql_est_step])
pipeline_run = exp.submit(pipeline)

When launching this, the logs for the container building service reveal:

FROM continuumio/miniconda3:4.4.10... etc.

Which indicates it's ignoring my FROM mcr.... statement in the Environment class I've associated with this Estimator, and my pip install fails.

Am I missing something? Is there a workaround?

azure-machine-learning-service
3 Answers

I can confirm that this is a bug on the AML Pipeline side. Specifically, the runconfig property environment.docker.base_dockerfile is not being passed through correctly in pipeline jobs. We are working on a fix. In the meantime, you can use the workaround from this thread of building the docker image first and specifying it with environment.docker.base_image (which is passed through correctly).

user_121916060

I found a workaround for now, which is to build your own Docker image. You can do this by using these options of the DockerSection of the Environment :

myenv.docker.base_image_registry.address = '<your_acr>.azurecr.io'
myenv.docker.base_image_registry.username = '<your_acr>'
myenv.docker.base_image_registry.password = '<your_acr_password>'
myenv.docker.base_image = '<your_acr>.azurecr.io/testimg:latest'

and use obviously whichever docker image you built and pushed to the container registry linked to the Azure Machine Learning Workspace.

To create the image, you would run something like this at the command line of a machine that can build a linux based container (like a Notebook VM):

docker build . -t <your_image_name>
# Tag it for upload
docker tag <your_image_name:latest <your_acr>.azurecr.io/<your_image_name>:latest
# Login to Azure
az login
# login to the container registry so that the push will work
az acr login --name <your_acr>
# push the image
docker push <your_acr>.azurecr.io/<your_image_name>:latest

Once the image is pushed, you should be able to get that working.

user_104355560

I also initially used EstimatorStep for custom images, but recently have figured out how to successfully pass Environment's first to RunConfiguration's, then to PythonScriptStep's. (example below)

Another workaround similar to your workaround would be to publish your custom docker image to Docker hub, then the param, docker_base_image becomes the URI, in our case mmlspark:0.16.

def get_environment(env_name, yml_path, user_managed_dependencies, enable_docker, docker_base_image):
    env = Environment(env_name)
    cd = CondaDependencies(yml_path)
    env.python.conda_dependencies = cd
    env.python.user_managed_dependencies = user_managed_dependencies
    env.docker.enabled = enable_docker
    env.docker.base_image = docker_base_image
    return env


spark_env = f.get_environment(env_name='spark_env',
                              yml_path=os.path.join(os.getcwd(), 'compute/aml_config/spark_compute_dependencies.yml'),
                              user_managed_dependencies=False, enable_docker=True,
                              docker_base_image='microsoft/mmlspark:0.16')

# use pyspark framework
spark_run_config = RunConfiguration(framework="pyspark")
spark_run_config.environment = spark_env

roll_step = PythonScriptStep(
    name='rolling window',
    script_name='roll.py',
    arguments=['--input_dir', joined_data,
                '--output_dir', rolled_data,
                '--script_dir', ".",
                '--min_date', '2015-06-30',
                '--pct_rank', 'True'],
    compute_target=compute_target_spark,
    inputs=[joined_data],
    outputs=[rolled_data],
    runconfig=spark_run_config,
    source_directory=os.path.join(os.getcwd(), 'compute', 'roll'),
    allow_reuse=pipeline_reuse
)

A couple of other points (that may be wrong):

  • PythonScriptStep is effectively a wrapper for ScriptRunConfig, which takes run_config as an argument
  • Estimator is a wrapper for ScriptRunConfig where RunConfig settings are made available as parameters
  • IMHO EstimatorStep shouldn't exist because it is better to define Env's and Steps separately instead of at the same time in one call.
user_38426100
Related