I want to create a cron job which reads some files from s3 , does some ETL and uploads/saves it backs to an s3 bucket. I have created and tested a cron job in aws triggered via an event. I want to hook it up to a task in fargate or something similar, preferably I want to run my ETL code in a docker container in aws as well. I have packaged my code in a container and i mount a volume in my local machine and run it successfully in local. it saves the file to my local machine. , but when i run this image/container application in aws. how do i save to file to an s3 bucket.
FROM python:3.8-slim-buster
WORKDIR /src
COPY src/requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD [ "python", "app.py"]
app.py
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve
...
# Plot metrics
plot_roc(y_test, y_score)
....
#save the file
plt.savefig(f"{images_dir}/precision_recall-cv-oversampling.png")