I have AWS s3 bucket where I am receiving multiple parquet files every minutes after performing some operation in AWS firehose. Now I have to make Real time sync of these files with GCP cloud storage bucket as we have multi cloud env and further process will be happening in GCP cloud. But I have problem that how can I do real time sync between two cloud buckets so that as soon as any file comes to AWS s3, same time it should come to GCP bucket as well. Any inputs please
1 Answers
If you literally mean updates happen at S3 and GCS atomically, that's not possible. The best you could do is have a job that gets notifications when updates complete at one, and initiate a copy to the other. You'd need to put some work into making the job robust regarding transient failures.
Related