I have a python script that I want to run on S3 files and send the output to another S3 bucket.
Now I could kick off an EC2 instance and interact using boto to do this and that's fine. But this doesn't seem to have an automatic way of shutting down the EC2 once the processes are complete (I'm going to be operating on about 100GB worth of data so I don't want to sit there and watch it).
The data pipelines of AWS seem attractive in that they scale appropriately and release resources when done. Which is great. But I can't seem to find a way to run a python script in a pipeline. The ShellCommandActivity seems closest but I'm not able to figure out how to set it up such that I can have the proper virtual environment built (with the appropriate packages, etc). Trying to figure out the best way to achieve this. Any help would be greatly appreciated