My main purpose: Easy way to collect information about different failure scenarios in SageMaker TrainingJob.
What do I use currently?
Sagemaker SKLearn Estimator(TrainingJobs are inside)
Where will my model train?
Different datasets. So, I need control and collect all information about all training processes and their final statuses on different datasets.
Which failure scenarios do I have?
There are plenty of them. I have create my own python Errors for them. For example: 1. There are labels only for one class. 2. Too small dataset(by my own criterions) 3. Missing data for crucial columns 4. e.t.c.
Where am I stuck?
After failed training I can't get own errors from training job response. All of them are ExecuteUserScriptError I can't pass my own info in FailureReason or ErrorMessage(always it's empty). I see which error was raised in CloudWatchLogs and TrainingJob Failure Traceback(from SagemakerNotebook). So, bad solution is parse CloudWatchLogs or /opt/ml/output/failure in case of failure with Lambda function.
Question: How to provide my own ErrorMessage or FailureReason in TrainingJob description?