I'm trying to find a way to re-direct the output as well as the errors to AWS S3 as a file inside docker. There are already some great answers in this link. Using the first answer given in the link I tried the following (outisde docker):
python3 train.py 2>&1 | aws s3 cp - s3://my_bucket_name/folder/output.log
This is working properly. The output.log file gets created in my s3 bucket as I intent to do. But when I put the same command as the CMD command inside Dockerfile it does nothing.
CMD python3 train.py 2>&1 | aws s3 cp - s3://my_bucket_name/folder/output.log
In fact docker kind of stuck and terminates after a while.
But, if I use the following code inside docker, the output gets created in the mount directory without any issue:
CMD python3 train.py > /mount/directory/output.log 2>&1
But I want the file uploaded to S3 live.
My use case: I'm trying to train a deep learning model in an EC2 instance, but I want to somehow get whatever happening on the console as a log file and store it on S3 live. From S3 whenever a log file is uploaded lambda triggers and it sends that log file to the localhost/another server for some processing.
Also, is there any way to show the output in the main console along with the file being uploaded in S3?
P.S. I don't have a software background. I'm a mathematician trying to get in the field of deep learning. So, if I've framed the question wrong or used wrong terminologies pardon me.