TensorFlow estimator is getting incorrect download input path

Viewed 78

I am trying to run a TensorFlow estimator in Sagemaker Studio. This has worked in the past but after saving an html file of my notebook, that path is now being appended to the directory I am providing to download the input data for the training session.

My code:

    # Set base model name that is used to save and load the model. Append a timestamp to it for uniqueness if batch tuning.
#Model Definition File
model_file = "Model.py"
#Input data saved as .npy
input_name = "inputdata.npy"
#Label data saved as .npy
label_name = "inputlabels.npy"
#Test data saved as .npy
test_name = "testdata.npy"
#Test labels saved as .npy
test_labels ="testlabels.npy"
#Path for trained model
bucket_dir = "s3://my-bucket"
#Bucket for trained model
bucket = "my-bucket"
model_name = "my_model"

model_name = model_name + ct.strftime("-%m%d%y-%H%M%S")
load_model_dir = os.path.join(bucket_dir,model_name)
data_dest = "/" + model_name + "/"+ input_name
print(logfile_name)
print(model_name)
print(load_model_dir)
print(data_dest)

#Define hyperparameters for hyperparameter tuning and set default values
shared_hyperparameters = {
    'bucket': bucket,
    'model_name': model_name,
    'model_dir':model_name,
    'sm_model_dir':model_name,
    'logfile_name': logfile_name,
    'train_data': input_name,
    'learning_rate': .001,
    'epochs': 100,
    'train': bucket_dir,
    'test': bucket_dir,
    'train_labels':label_name,
    'test_data': test_name,
    'test_labels': test_labels
}
...

aws_estimator = TensorFlow(
    entry_point= model_file, #Model definition .py file
    bucket = bucket,
    role= role,
    instance_count=1,
    instance_type="ml.m5.2xlarge",
    framework_version="2.1.0",
    py_version="py3",
    distribution={"parameter_server": {"enabled": True}},
    hyperparameters = shared_hyperparameters,
    metric_definitions = metric_definitions,
    log="All",
    my_name = model_name,
    log_name = logfile_name,
    train_data = input_name,
    train_labels=label_name,
)
history = aws_estimator.fit(bucket_dir)

This results in the following error:

UnexpectedStatusException: Error for Training job tensorflow-training-2021-11-19-22-51-12-360: Failed. Reason: ClientError: Data download failed:S3 key: s3://my-bucket/s3://my-bucket/model_dir/my-notebook.html has an illegal char sub-sequence '//' in it

I am not sure why the path to the HTML file is being appended onto the bucket_dir when it wasn't before. I saw a similar problem on the AWS forums, but no helpful response was provided. I have tried printing out what the SM_CHANNEL_TRAINING environment variable is before and after training, and it is None.

2 Answers

I'm not sure what your issues is, but what's weird is that your passing lots of non existing parameters to the TensorFlow estimator object (e.g., 'bucket').
To get started I recommend to clean up things: see list of known parameters here (also check the base classes: Framework and EstimatorBase), remove the non existing parameters, and try again.

For some reason, this problem was resolved when I switched to running an instance with TensorFlow 2.3 instead of 2.7.

Related