How to split Azure ML pipeline steps to debug

Question

I have created an Azure ML pipeline with different steps (data preprocess, train, validation ...). And for pass data from one step to the next I have used the PipelineData object.

Example passing the model from train step to validate one:

    # Create a PipelineData to pass model from train to register
    model_path = PipelineData('model')

    # Step 2
    train_step = PythonScriptStep(
        name = 'Train the Model',
        script_name = 'TrainStepScript.py',
        source_directory = source_folder,
        arguments = ['--training_data', prepped_data,
                     '--model_name', model_name,
                     '--model_path', model_path],
        outputs = [model_path],
        compute_target = compute_target,
        runconfig = aml_run_config,
        allow_reuse = True
    )

    # Step 3
    third_step = PythonScriptStep(
        name = 'Evaluate & register the Model',
        script_name = 'ValidateStepScript.py',
        source_directory = source_folder,
        arguments = ['--model_name', model_name,
                     '--model_path', model_path],
        inputs = [model_path],
        compute_target = compute_target,
        runconfig = aml_run_config,
        allow_reuse = True
    )

Now for debugging and development purposes I want to create a script to run separately the different steps using a ScriptRunConfig (with the same environment and arguments of the StepScript in the pipeline). But the problem is I don't know how to simulate the data input/output of each step, because the DataPipeline object is not working for this purpose.

Just for clarification, my goal is to NOT modify the original pipeline StepScripts, so I can use them after debugging in the final pipeline. To sum up, my question is: how can I emulate the DataPipeline object (if possible) in this case?

Example of what I'm trying to build:

# Passing in some way the model path (from local)
model_path = PipelineData('model')

# Create a script config for validate step
validate_script_config = ScriptRunConfig(
    source_directory = source_folder,
    script = 'ValidateStepScript.py',
    arguments = ['--model_name', model_name,
                 '--model_path', model_path],
    environment = experiment_env,
    docker_runtime_config = DockerConfiguration(use_docker=True)
)

experiment = Experiment(workspace=ws, name=experiment_name)
data_run = experiment.submit(config=data_script_config)

TadepalliSairam · Answer

We can download the output of the model in a repository and make them as the source file for the later steps as required. The below code block can be incorporated in the same pipeline which is being used now.

To download the model output:

train_step = pipeline_run1.find_step_run('train.py')

if train_step:
    train_step_obj = train_step[0] 
    train_step_obj.get_output_data('processed_data1').download("./outputs") # download the output to current directory

after downloading the model, then use that as the parent source directory in source_directory

from azureml.core import ScriptRunConfig, Experiment
   # create or load an experiment
   experiment = Experiment(workspace, 'MyExperiment')
   # create or retrieve a compute target
   cluster = workspace.compute_targets['MyCluster']
   # create or retrieve an environment
   env = Environment.get(ws, name='MyEnvironment')
   # configure and submit your training run
   config = ScriptRunConfig(source_directory='.',
                            command=['python', 'train.py'],
                            compute_target=cluster,
                            environment=env)
   script_run = experiment.submit(config)

How to split Azure ML pipeline steps to debug

1 Answers