Is there a way to include custom Regression Metrics in ModelQualityMonitor in AWS sagemaker?

user_15236333 0
Viewed 165

I have successfully initialized a ModelQualityMonitor object. Then I created a monitoring schedule using the CreateMonitoringSchedule API! In the background sagemaker runs two processing jobs which merges the ground truth data with the collected endpoint data and then analyzes and creates the predefined regression metrics: https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-metrics.html

Unfortunately, I am missing the MAPE (Mean Absolute Percentage Error) in the metrics, and would like to create this with in the future (also in CloudWatch).

Sagemaker provides the following functionalities:

  • Preprocessing and Postprocessing: In addition to using the built-in mechanisms, you can extend the code with the preprocessing and postprocessing scripts.
  • Bring Your Own Containers: Amazon SageMaker Model Monitor provides a prebuilt container with ability to analyze the data captured from endpoints for tabular datasets. If you would like to bring your own container, Model Monitor provides extension points which you can leverage.
  • CloudWatch Metrics for Bring Your Own Containers

Those points are documented on this site: https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-custom-monitoring-schedules.html

How exactly can I achieve my target of including MAPE with the above points?

Here is a code snippet of my current implementation:

from sagemaker.model_monitor.model_monitoring import ModelQualityMonitor
from sagemaker.model_monitor import EndpointInput
from sagemaker.model_monitor.dataset_format import DatasetFormat

# Create the model quality monitoring object
MQM = ModelQualityMonitor(
    role=role,
    instance_count=1,
    instance_type="ml.m5.large",
    volume_size_in_gb=20,
    max_runtime_in_seconds=1800,
    sagemaker_session=sagemaker_session,
)

# suggest a baseline
job = MQM.suggest_baseline(
    job_name=baseline_job_name,
    baseline_dataset="./baseline.csv",
    dataset_format=DatasetFormat.csv(header=True),
    output_s3_uri=baseline_results_uri,
    problem_type="Regression",
    inference_attribute="predicted_price",
    ground_truth_attribute="price",
)
job.wait(logs=False)
baseline_job = MQM.latest_baselining_job

# create a monitoring schedule
endpointInput = EndpointInput(
    endpoint_name="dev-TestEndpoint",
    destination="/opt/ml/processing/input_data",
    inference_attribute="$.data.predicted_price"
)
MQM.create_monitoring_schedule(
    monitor_schedule_name="DS-Schedule",
    endpoint_input=endpointInput,
    output_s3_uri=baseline_results_uri,
    constraints=baseline_job.suggested_constraints(),
    problem_type="Regression",
    ground_truth_input=ground_truth_upload_path,
    schedule_cron_expression="cron(0 * ? * * *)", # hourly
    enable_cloudwatch_metrics=True
)
amazon-sagemaker boto3 amazon-web-services monitoring python
1 Answers

Amazon SageMaker model monitor only supports metrics that are defined here out of the box. If you need to include another metric such as MAPE (Mean Absolute Percentage Error) in your case, you will have to rely on BYOC approach, note that with this approach you cannot "add" a metric to the available list, unfortunately you will have to implement the entire suite of metrics yourself. I understand this is not ideal for customers, I'd encourage you to reach out to your AWS account manager to create a request to add MAPE (Mean Absolute Percentage Error) as a supported metric in the long run. I've made a note of it as well and will rely it back to the team.

In the meantime, you can find examples on how to BYOC here.

I work for AWS but my opinions are my own.

Thanks, Raghu

user_64115480
Related