How to import custom functions on my experiment script for Azure ML?

user_6622571 0
Viewed 428

I can successfully submit an experiment to processing on a remote compute target on Azure ML.

In my notebook, for submitting the experiment, I have:

# estimator
estimator = Estimator(
    source_directory='scripts',
    entry_script='exp01.py',
    compute_target='pc2',
    conda_packages=['scikit-learn'],
    inputs=[data.as_named_input('my_dataset')],
    )

# Submit
exp = Experiment(workspace=ws, name='my_exp')

# Run the experiment based on the estimator
run = exp.submit(config=estimator)
RunDetails(run).show()
run.wait_for_completion(show_output=True)

However, in order to keep things clean, I want to define my general use functions on an auxiliary script, so the first will import it.

On my script experiment file exp01.py, I wanted:

import custom_functions as custom

# azure experiment start
run = Run.get_context()

# the data from azure datasets/datastorage
df = run.input_datasets['my_dataset'].to_pandas_dataframe()

# prepare data
df_transformed = custom.prepare_data(df)

# split data
X_train, X_test, y_train, y_test = custom.split_data(df_transformed)

# run my models.....
model_name = 'RF'
model = custom.model_x(model_name, a_lot_of_args)

# log the results
run.log(model_name, results)

# azure finish
run.complete()

The thing is: Azure wont let me import the custom_functions.py.

How are you doing it?

azure-machine-learning-service scikit-learn azure python-3.x
1 Answers

TL;DR any files you put inside the source_directory in your case, scripts will be available to the Estimator.

To make this happen, simply create a file called custom_functions.py in the scripts folder that contains your prepare_data(), split_data(), model_x() functions.

I also recommend that you include only exactly what you need in the source_directory folder and make distinct folders for each Estimator because:

  • the entire folder's contents will be uploaded when you use a remote compute_target, and
  • when you started using ML Pipeilnes (which are awesome), PythonScriptSteps allow_reuse parameter will look to see if any files in the source_directory have changed when determining if the step needs to run again or not.

Lastly, when you want to share general utility functions across PythonScriptSteps or Estimators without having to copy and paste code, that's when you might want to consider creating a custom python package.

user_38426100
Related