As mentioned in question, I keep getting UnexpectedStatusException: Error for HyperParameterTuning job xgboost-211***-1631: Failed. Reason: No training job succeeded after 5 attempts. For additional details, please take a look at the training job failures by listing training jobs for the hyperparameter tuning job.
I looked into parameter ranges based on https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html to make sure that ranges are good and they seem to be ok. Data is definitely good because I can train the model but can't tune it.
Here is the code I am using :
import sagemaker
import boto3
import numpy as np # For matrix operations and numerical processing
import pandas as pd # For munging tabular data
import os
from sagemaker.tuner import IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner
from sagemaker.session import TrainingInput
from sagemaker.debugger import Rule, rule_configs
region = boto3.Session().region_name
smclient = boto3.Session().client('sagemaker')
role = sagemaker.get_execution_role()
s3_output_location='s3://{}/{}/{}'.format(bucket, prefix, 'output')
container=sagemaker.image_uris.retrieve("xgboost", region, "latest")
xgb_model=sagemaker.estimator.Estimator(
image_uri=container,
role=role,
instance_count=1,
instance_type='ml.m4.xlarge',
volume_size=5,
output_path=s3_output_location,
sagemaker_session=sagemaker.Session(),
rules=[Rule.sagemaker(rule_configs.create_xgboost_report())]
)
xgb_model.set_hyperparameters(
max_depth = 5,
eta = 0.2,
gamma = 4,
min_child_weight = 6,
subsample = 0.7,
objective = "binary:logistic",
num_round = 10
)
hyperparameter_ranges = {'eta': ContinuousParameter(0.1, 0.5),
'min_child_weight': ContinuousParameter(1, 10),
'alpha': ContinuousParameter(0, 3),
'max_depth': IntegerParameter(0, 4)}
objective_metric_name = 'validation:auc'
tuner = HyperparameterTuner(xgb_model,
objective_metric_name,
hyperparameter_ranges,
max_jobs=60,
max_parallel_jobs=6)
train_input = TrainingInput(
"s3://{}/{}/{}".format(bucket, prefix, "train/train.csv"), content_type="csv"
)
validation_input = TrainingInput(
"s3://{}/{}/{}".format(bucket, prefix, "validate/validation.csv"), content_type="csv"
)
tuner.fit({"train": train_input, "validation": validation_input}, include_cls_metadata=False)
This is the error I get
> --------------------------------------------------------------------------- UnexpectedStatusException Traceback (most recent call
> last) <ipython-input-2-7824ad80a8bb> in <module>
> 62 )
> 63
> ---> 64 tuner.fit({"train": train_input, "validation": validation_input}, include_cls_metadata=False)
>
> ~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py
> in fit(self, inputs, job_name, include_cls_metadata, estimator_kwargs,
> wait, **kwargs)
> 449
> 450 if wait:
> --> 451 self.latest_tuning_job.wait()
> 452
> 453 def _fit_with_estimator(self, inputs, job_name, include_cls_metadata, **kwargs):
>
> ~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py
> in wait(self) 1595 def wait(self): 1596
> """Placeholder docstring."""
> -> 1597 self.sagemaker_session.wait_for_tuning_job(self.name) 1598 1599
>
> ~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py
> in wait_for_tuning_job(self, job, poll) 3253 """ 3254
> desc = _wait_until(lambda: _tuning_job_status(self.sagemaker_client,
> job), poll)
> -> 3255 self._check_job_status(job, desc, "HyperParameterTuningJobStatus") 3256 return desc 3257
>
> ~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py
> in _check_job_status(self, job, desc, status_key_name) 3336
> ), 3337 allowed_statuses=["Completed", "Stopped"],
> -> 3338 actual_status=status, 3339 ) 3340
>
> UnexpectedStatusException: Error for HyperParameterTuning job
> xgboost-211XXX-1641: Failed. Reason: No training job succeeded after 5
> attempts. For additional details, please take a look at the training
> job failures by listing training jobs for the hyperparameter tuning
> job.
Thank you in advance,
Sam