How to deploy sagemaker.workflow.pipeline.Pipeline?

user_11844406 0
Viewed 313

I have a sagemaker.workflow.pipeline.Pipeline which contains multiple sagemaker.workflow.steps.ProcessingStep and each ProcessingStep contains sagemaker.processing.ScriptProcessor.

The current pipeline graph look like the below shown image. It will take data from multiple sources from S3, process it and create a final dataset using the data from previous steps.

enter image description here

As the Pipeline object doesn't support .deploy method, how to deploy this pipeline?

While inference/scoring, When we receive a raw data(single row for each source), how to trigger the pipeline?

or Sagemaker Pipeline is designed for only data processing and model training on huge/batch data? Not for the inference with the single data point?

amazon-sagemaker amazon-web-services python
1 Answers

As the Pipeline object doesn't support .deploy method, how to deploy this pipeline?

Pipeline does not have a .deploy() method, no

Use pipeline.upsert(role_arn='...') to create/update the pipeline definition to SageMaker, then call pipeline.start() . Docs here

While inference/scoring, When we receive a raw data(single row for each source), how to trigger the pipeline?

There are actually two types of pipelines in SageMaker. Model Building Pipelines (which you have in your question), and Serial Inference Pipelines, which are used for Inference. AWS definitely should have called the former "workflows"

You can use a model building pipeline to setup a serial inference pipeline

To do pre-processing in a serial inference pipeline, you want to train an encoder/estimator (such as SKLearn) and save its model. Then train a learning algorithm, and save its model, then create a PipelineModel using both models

user_2234780
Related