I am trying to run a TensorFlow estimator in Sagemaker Studio. This has worked in the past but after saving an html file of my notebook, that path is now being appended to the directory I am providing to download the input data for the training session.
My code:
# Set base model name that is used to save and load the model. Append a timestamp to it for uniqueness if batch tuning.
#Model Definition File
model_file = "Model.py"
#Input data saved as .npy
input_name = "inputdata.npy"
#Label data saved as .npy
label_name = "inputlabels.npy"
#Test data saved as .npy
test_name = "testdata.npy"
#Test labels saved as .npy
test_labels ="testlabels.npy"
#Path for trained model
bucket_dir = "s3://my-bucket"
#Bucket for trained model
bucket = "my-bucket"
model_name = "my_model"
model_name = model_name + ct.strftime("-%m%d%y-%H%M%S")
load_model_dir = os.path.join(bucket_dir,model_name)
data_dest = "/" + model_name + "/"+ input_name
print(logfile_name)
print(model_name)
print(load_model_dir)
print(data_dest)
#Define hyperparameters for hyperparameter tuning and set default values
shared_hyperparameters = {
'bucket': bucket,
'model_name': model_name,
'model_dir':model_name,
'sm_model_dir':model_name,
'logfile_name': logfile_name,
'train_data': input_name,
'learning_rate': .001,
'epochs': 100,
'train': bucket_dir,
'test': bucket_dir,
'train_labels':label_name,
'test_data': test_name,
'test_labels': test_labels
}
...
aws_estimator = TensorFlow(
entry_point= model_file, #Model definition .py file
bucket = bucket,
role= role,
instance_count=1,
instance_type="ml.m5.2xlarge",
framework_version="2.1.0",
py_version="py3",
distribution={"parameter_server": {"enabled": True}},
hyperparameters = shared_hyperparameters,
metric_definitions = metric_definitions,
log="All",
my_name = model_name,
log_name = logfile_name,
train_data = input_name,
train_labels=label_name,
)
history = aws_estimator.fit(bucket_dir)
This results in the following error:
UnexpectedStatusException: Error for Training job tensorflow-training-2021-11-19-22-51-12-360: Failed. Reason: ClientError: Data download failed:S3 key: s3://my-bucket/s3://my-bucket/model_dir/my-notebook.html has an illegal char sub-sequence '//' in it
I am not sure why the path to the HTML file is being appended onto the bucket_dir when it wasn't before. I saw a similar problem on the AWS forums, but no helpful response was provided. I have tried printing out what the SM_CHANNEL_TRAINING environment variable is before and after training, and it is None.