AzureML Dataset.File.from_files creation extremely slow even with 4 files

user_5080195 0
Viewed 989

I have a few thousand of video files in my BlobStorage, which I set it as a datastore. This blob storage receives new files every night and I need to split the data and register each split as a new version of AzureML Dataset.

This is how I do the data split, simply getting the blob paths and splitting them.

container_client = ContainerClient.from_connection_string(AZ_CONN_STR,'keymoments-clips')
blobs = container_client.list_blobs('soccer')
blobs = map(lambda x: Path(x['name']), blobs)
train_set, test_set = get_train_test(blobs, 0.75, 3, class_subset={'goal', 'hitWoodwork', 'penalty', 'redCard', 'contentiousRefereeDecision'})
valid_set, test_set = split_data(test_set, 0.5, 3)

train_set, test_set, valid_set are just nx2 numpy arrays containing blob storage path and class.

Here is when I try to create a new version of my Dataset:

datastore = Datastore.get(workspace, 'clips_datastore')

dataset_train = Dataset.File.from_files([(datastore, b) for b, _ in train_set[:4]], validate=True, partition_format='**/{class_label}/*.mp4')
dataset_train.register(workspace, 'train_video_clips', create_new_version=True)

How is it possible that the Dataset creation seems to hang for an indefinite time even with only 4 paths? I saw in the doc that providing a list of Tuple[datastore, path] is perfectly fine. Do you know why?

Thanks

azure-sdk-python azure-machine-learning-studio azure-machine-learning-service
3 Answers

Do you have your Azure Machine Learning Workspace and your Azure Storage Account in different Azure Regions? If that's true, latency may be a contributing factor with validate=True.

user_66409800

I'd be interested to see what happens if you run the dataset creation code twice in the same notebook/script. Is it faster the second time? I ask because it might be an issue with the .NET core runtime startup (which would only happen on the first time you run the code)

EDIT 9/16/20

While it doesn't seem to make sense that .NET core invoked when not data is moving, is suspect it is the validate=True part of the param that requires that all the data be inspected (which can computationally expensive). I'd be interested to see what happens if that param is False

user_38426100

Another possibility may be slowness in the way datastore paths are resolved. This is an area where improvements are being worked on.

As an experiment, could you try creating the dataset using a url instead of datastore? Let us know if that makes a difference to performance, and whether it can unblock your current issue in the short term.

Something like this:

dataset_train = Dataset.File.from_files(path="https://bloburl/**/*.mp4?accesstoken", validate=True, partition_format='**/{class_label}/*.mp4')
dataset_train.register(workspace, 'train_video_clips', create_new_version=True)
user_23412200
Related