Naming a Sagemaker Processing job using Sagemaker Pipelines ProcessingStep

user_17752459 0
Viewed 417

I am running a Sagemaker Pipeline with the current processor:

from sagemaker.sklearn.processing import SKLearnProcessor


framework_version = "0.23-1"

sklearn_processor = SKLearnProcessor(
    framework_version=framework_version,
    instance_type=processing_instance_type,
    instance_count=processing_instance_count,
    base_job_name="pre-processing-job-name",
    role=role
)

and the processing step is:

from sagemaker.processing import ProcessingInput, ProcessingOutput
from sagemaker.workflow.steps import ProcessingStep


step_process = ProcessingStep(
    name="AbaloneProcess",
    processor=sklearn_processor,
    inputs=[
        ProcessingInput(source=input_data, destination="/opt/ml/processing/input"),
    ],
    outputs=[
        ProcessingOutput(output_name="train", source="/opt/ml/processing/train"),
        ProcessingOutput(output_name="validation", source="/opt/ml/processing/validation"),
        ProcessingOutput(output_name="test", source="/opt/ml/processing/test"),
    ],
    code="abalone/preprocessing.py",
)

It looks like the base_job_name does nothing, because the processing job that is created is pipelines-o6e2jn38g05j-AbaloneProcess-nc2OlXF8jA.

I want the processing job name to be defined manually. Does Sagemaker pipelines support this? I seem to be going around in circles.

data-science amazon-sagemaker machine-learning amazon-web-services
1 Answers

Pipelines sets the processing job name to preserve caching behavior.

An override could be added here to allow manually defining job names, but that has been intentionally left out. Because SageMaker job names must be unique, this would become a sharp edge for users who statically define the name without realizing that any pipeline execution after the first will fail with ResourceAlreadyExists.

I'd suggest filing an issue explaining the usecase and see if the team picks it up.

user_8956150
Related