How to pass dependency files to sagemaker SKLearnProcessor and use it in Pipeline?

user_8353711 0
Viewed 2139

I need to import function from different python scripts, which will used inside preprocessing.py file. I was not able to find a way to pass the dependent files to SKLearnProcessor Object, due to which I am getting ModuleNotFoundError.

Code:

from sagemaker.sklearn.processing import SKLearnProcessor
from sagemaker.processing import ProcessingInput, ProcessingOutput

sklearn_processor = SKLearnProcessor(framework_version='0.20.0',
                                     role=role,
                                     instance_type='ml.m5.xlarge',
                                     instance_count=1)


sklearn_processor.run(code='preprocessing.py',
                      inputs=[ProcessingInput(
                        source=input_data,
                        destination='/opt/ml/processing/input')],
                      outputs=[ProcessingOutput(output_name='train_data',
                                                source='/opt/ml/processing/train'),
                               ProcessingOutput(output_name='test_data',
                                                source='/opt/ml/processing/test')],
                      arguments=['--train-test-split-ratio', '0.2']
                     )

I would like to pass, dependent_files = ['file1.py', 'file2.py', 'requirements.txt']. So, that preprocessing.py have access to all the dependent modules.

And also need to install libraries from requirements.txt file.

Can you share any work around or a right way to do this?

Update-25-11-2021:

Q1.(Answered but looking to solve using FrameworkProcessor)

Here, the get_run_args function, is handling dependencies, source_dir and code parameters by using FrameworkProcessor. Is there any way that we can set this parameters from ScriptProcessor or SKLearnProcessor or any other Processor to set them?

Q2.

Can you also please show some reference to use our Processor as sagemaker.workflow.steps.ProcessingStep and then use in sagemaker.workflow.pipeline.Pipeline?

For having Pipeline, do we need sagemaker-project as mandatory or can we create Pipeline directly without any Sagemaker-Project?

amazon-sagemaker scikit-learn amazon-web-services python
2 Answers

There are a couple of options for you to accomplish that.

One that is really simple is adding all additional files to a folder, example:

.
├── my_package
│   ├── file1.py
│   ├── file2.py
│   └── requirements.txt
└── preprocessing.py

Then send this entire folder as another input under the same /opt/ml/processing/input/code/, example:

from sagemaker.sklearn.processing import SKLearnProcessor
from sagemaker.processing import ProcessingInput, ProcessingOutput

sklearn_processor = SKLearnProcessor(
    framework_version="0.20.0",
    role=role,
    instance_type="ml.m5.xlarge",
    instance_count=1,
)

sklearn_processor.run(
    code="preprocessing.py",  # <- this gets uploaded as /opt/ml/processing/input/code/preprocessing.py
    inputs=[
        ProcessingInput(source=input_data, destination='/opt/ml/processing/input'),
        # Send my_package as /opt/ml/processing/input/code/my_package/
        ProcessingInput(source='my_package/', destination="/opt/ml/processing/input/code/my_package/")
    ],
    outputs=[
        ProcessingOutput(output_name="train_data", source="/opt/ml/processing/train"),
        ProcessingOutput(output_name="test_data", source="/opt/ml/processing/test"),
    ],
    arguments=["--train-test-split-ratio", "0.2"],
)

What happens is that sagemaker-python-sdk is going to put your argument code="preprocessing.py" under /opt/ml/processing/input/code/ and you will have my_package/ under the same directory.

Edit:

For the requirements.txt, you can add to your preprocessing.py:

import sys
import subprocess

subprocess.check_call([
    sys.executable, "-m", "pip", "install", "-r",
    "/opt/ml/processing/input/code/my_package/requirements.txt",
])
user_23738820

This isn't supported in SKLearnProcessor. You'd need to package your dependencies in docker image and create a custom Processor (e.g. a ScriptProcessor with the image_uri of the docker image you created.)

user_8458430
Related