Syncing remote folders from several machines to one AWS instance

user_10061137 0
Viewed 143

I have 3 AWS P instances processing some heavy stuff and saving results to relevant /home/user/folder
Also I have a main server with the same folder where I want to collect results from those 3 instances
Each instance works on its own part of the whole task, their results in sub folders not overlapping

Instances are 2 TB each, so I would like to get results from each instance as soon as they appear
This way when its job is done, I won't spend half a day copying results to the main server

I think one way of solving this is running something like this on each instance:

*/30 * * * * rsync /home/user/folder ubuntu@1.1.1.1:/home/user/folder

Are there any other more smart ways of achieving same results given that all of instances are AWS?
I also thought about (1) detachable storage and (2) storing on S3 but being new to AWS I might overlook some hidden pitfalls in such workflows, especially when it comes to terabytes of data and expensive instances.

How do you collect processed data from remote instances?

amazon-web-services amazon-ec2 rsync amazon-s3 cron
2 Answers

My thoughts on some of the options mentioned in OP and comments, as well as some other ones I thought of:

  1. EFS: create an EFS and mount it as an NFS drive on all the instances. It's the easiest but probably costs the most.
  2. s3fs: have all the instances mount the same S3 bucket using s3fs. This is likely the most inexpensive solution. You also don't need to worry about running out of disk space. The downside is that the performance is not going to be that good compared to mounted NFS drives.
  3. EBS volumes: attach an EBS volume to each worker instance for them to write the results to. When they are done, detach the volumes and attach them to the main server. This will be the fastest and still cheaper than EFS. If you can't or won't do all the detaching/attaching manually you'll need to write some scripts.
  4. Old school NFS shares: there is nothing wrong with a plain vanilla NFS setup without any of those fancy AWS acronyms. :-)
user_15263470

I would consider using rclone tool, which can be easy configured for the shared S3 bucket. Just be aware about copy/sync mode. It can rich up to several Gigabit throughput depending on your instance type.

Link for the project: rclone.org

user_55051640
Related