Can we use data directly from RDS or df as a data source for training job in Sagemaker, rather than pulling it from from s3 or EFS?

user_10838658 0
Viewed 160

I am using Sagemaker platform for model development and deployment. Data is read from RDS tables and then spitted to train and test df. To create the training job in Sagemaker, I found that it takes data source only as s3 and EFS. For that I need to keep train and test data back to s3, which is repeating the data storing process in RDS and s3. I would want to directly pass the df from RDS as a parameter in tarining job code. Is there any way we can pass df in fit method

    image="581132636225.dkr.ecr.ap-south-1.amazonaws.com/sagemaker-ols-model:latest"
    model_output_folder = "model-output"
    print(image)
    tree = sagemaker.estimator.Estimator(
        image,
        role,
        1,
        "ml.c4.2xlarge",
        output_path="s3://{}/{}".format(sess.default_bucket(), model_output_folder),
        sagemaker_session=sess,
    )

**tree.fit({'train': "s3_path_having_test_data"}, wait=True)**
data-science amazon-sagemaker amazon-web-services python
1 Answers

The training data must be read from Amazon S3, Amazon EFS or Amazon FSx for Lustre.
One advantage of this is being able to reproduce your training results later on, as the input data is frozen in time (unless deleted), as apposed to a live DB.

Typical code:

train_df.to_csv("train.csv", header=False, index=False)
boto3.Session().resource("s3").Bucket(bucket).Object(
    os.path.join(prefix, "train/train.csv")
).upload_file("train.csv")
s3_path_having_test_data = "s3://{}/{}/train".format(bucket, prefix)

tree.fit({'train': "s3_path_having_test_data"}, wait=True)
user_1219560
Related