Backup GCS bucket to an archive-class GCS bucket

user_971626 0
Viewed 398

The GCS Transfer Service is a nice solution for regular transfer of data. What would the appropriate settings be to sync a bucket to another, where the destination bucket has the archive storage class?

One of my concerns is possible expensive operations on archive class objects, if the service is checking for identical objects. If, to avoid these checks, I just copy files daily that are new within the last 24 hours, how accurate is that timing -- can I miss a file due to timing jitter of several minutes on job start?

google-cloud-storage
1 Answers

While "the appropriate settings" would completely depend on your business needs. A good approach could be to use a Cloud Function triggered on create/finalize on your standard bucket that creates a copy of that object in your second bucket.

You can use this function in python as a guide:

from google.cloud import storage
from google.cloud.storage.blob import Blob

def hello_gcs(event, context):
    client = storage.Client()
    source_bucket = client.get_bucket("my_source_bucket)
    dest_bucket = client.get_bucket("my_destination_bucket")
    filename = str(event['id']).rsplit('/',1)[0]
    blob = Blob.from_string("gs://" + filename)
    source_bucket.copy_blob(blob,dest_bucket)
user_115049090
Related