Mounting a FileDatasets in Azure ML Services

user_12201890 0
Viewed 2898

I am facing a problem with the Dataset module in Azure Machine Learning Services. I created a FileDataset with a bunch of images to train a model in TensorFlow. I’m mounting the dataset in the target compute and then passing the mounting point to the train script as described in the sample notebook we have on GitHub.

I tried two approaches: to pass the path as an script parameter (as suggested on GitHub) and as a named input, but none of them seem to pass the mounting point correctly. Anyone does know which is the correct way to make it work? (I can make it work with Data Sources btw)

As script parameter

script_params = {
    '--data-folder': dset.as_named_input('dogscats_train').as_mount('tmp/dataset'),
} 

src = TensorFlow(source_directory =  r'Tensorflow',
              framework_version = '1.13',
              entry_script = 'train.py',
              script_params=script_params,
              compute_target='amlcompute', 
              vm_size='Standard_NC6', 
              use_gpu = True, 
              pip_packages = ['matplotlib', 'pillow', 'numpy', 'azureml-sdk'])

Mounted path:

/mnt/batch/tasks/shared/LS_root/jobs/aa-ml-aml-workspace/azureml/cats-vs-dogs-tensorflow_1570799752_014bea9f/mounts/workspaceblobstore/azureml/cats-vs-dogs-tensorflow_1570799752_014bea9f/tmp/dataset

Actual path received in the script:

/tmp/dataset

As named input

src = TensorFlow(source_directory =  r'Tensorflow',
              framework_version = '1.13',
              entry_script = 'train.py',
              inputs=[dset.as_named_input('dogscats_train')],
              compute_target='amlcompute', 
              vm_size='Standard_NC6', 
              use_gpu = True, 
              pip_packages = ['matplotlib', 'pillow', 'numpy', 'azureml-sdk'])

Mounted Path:

/mnt/batch/tasks/shared/LS_root/jobs/aa-ml-aml-workspace/azureml/cats-vs-dogs-tensorflow_1570804147_39168dcf/mounts/workspaceblobstore

Path retrieved by run.input_datasets['dogscats_train'].mount('tmp/dataset').mount_point:

/mnt/batch/tasks/shared/LS_root/jobs/aa-ml-aml-workspace/azureml/cats-vs-dogs-tensorflow_1570804147_39168dcf/mounts/workspaceblobstore/azureml/cats-vs-dogs-tensorflow_1570804147_39168dcf/tmp/dataset

azure-machine-learning-service tensorflow azure
1 Answers

In both approaches, the path you passed to as_mount and mount is a relative path, that is, it didn't start with a leading /, this will cause the dataset to be mounted relative to the current working directory, which in your case is /mnt/batch/tasks/shared/LS_root/jobs/aa-ml-aml-workspace/azureml/cats-vs-dogs-tensorflow_1570799752_014bea9f/mounts/workspaceblobstore/azureml/cats-vs-dogs-tensorflow_1570799752_014bea9f.

If you would like to mount to /tmp/dataset, you will need to do dset.as_named_input('dogscats_train').as_mount('/tmp/dataset'), that is, you need to explicitly add a / to the target path to make the path an absolute path.

I have created a gist to show the effect of relative path vs absolute path.

submit.py: Submits a TensorFlow run

so.py: compares the two arguments, the first argument is the relative path, the second is the absolute path.

driver_log_partial.txt: is a partial output of the driver log you see in the run details page.

Now in regards to the name you passed to as_named_input. That name is then made available as an environment variable whose value is the same as the script argument's value, which is the path where the dataset is mounted to. The name is also made available as a key to run.input_datasets where the value is also the path where the dataset is mounted to. In other words, there are three ways to access the mounted path, you can choose a one that best suits your code.

user_88779820
Related