GCP cloud function python - GCS copy files - duplicate files

user_6885516 0
Viewed 2287

Im trying to copy files from GCS to a different location. But I need to it in real time with Cloud function. I have created a function and its working. But the issue is, the file copied multiple times with multiple folders.

EG:

source file path: gs://logbucket/mylog/2020/07/22/log.csv

Expected Target: gs://logbucket/hivelog/2020/07/22/log.csv

My code:

from google.cloud import storage

def hello_gcs_generic(data, context):
    sourcebucket=format(data['bucket'])
    source_file=format(data['name'])
    year = source_file.split("/")[1]
    month = source_file.split("/")[2]
    day = source_file.split("/")[3]
    filename=source_file.split("/")[4]
    print(year)
    print(month)
    print(day)
    print(filename)
    print(sourcebucket)
    print(source_file)


    storage_client = storage.Client()

    source_bucket = storage_client.bucket(sourcebucket)
    source_blob = source_bucket.blob(source_file)
    destination_bucket = storage_client.bucket(sourcebucket)
    destination_blob_name = 'hivelog/year='+year+'/month='+month+'/day='+day+'/'+filename


    blob_copy = source_bucket.copy_blob(
        source_blob, destination_bucket, destination_blob_name
    )
    blob.delete()
    print(
        "Blob {} in bucket {} copied to blob {} in bucket {}.".format(
            source_blob.name,
            source_bucket.name,
            blob_copy.name,
            destination_bucket.name,
        )
    )

Output:

enter image description here

You can see this year=year=2020 how this comes? Also inside this I have folders like year=year=2020/month=month=07/

Im not able to fix this.

google-cloud-functions google-cloud-platform python
3 Answers

As an educated guess, the source paths you're copying from are really of the format

/foo/year=2020/month=42/...

so when you split by slashes, you get

foo
year=2020
month=42

and when recomposing those components, you again add another year=/month=/... prefix in

destination_blob_name = 'hivelog/year='+year+'/month='+month+ ...

and there you have it; year=year=year= after 3 iterations...

Are you also sure you're not also iterating over the files you've already copied? That would also cause this.

AKX_516850

You're writing to the same bucket that you're trying to copy from:

destination_bucket = storage_client.bucket(sourcebucket)

Every time you add a new file to the bucket, it's triggering the Cloud Function again.

You either need to use two different buckets, or add a conditional based on the first part of the path:

top_level_directory = source_file.split("/")[0]
if top_level_directory == "mylog":
    # Do the copying
elif top_level_directory == "hivelog":
    # This is a file created by the function, do nothing
else:
    # We weren't expecting this top level directory
user_3280360 
import os
import gcsfs


def hello_gcs_generic(data, context):
    fs = gcsfs.GCSFileSystem(project="Project_Name", token=os.getenv("GOOGLE_APPLICATION_CREDENTIALS"))
    source_filepath = f"{data['bucket']}/{data['name']}"
    destination_filepath = source_filepath.replace("mylog","hivelog")
    fs.cp(source_filepath,destination_filepath)
    print(f"Blob {data['name']} in bucket {data['bucket']} copied to hivelog")

This should give you a head start on what you're trying to accomplish. Replace Project_Name with the name of your GCP Project where the storage buckets are located.

Also assumes you have service account credentials in a json file set with the environment variable GOOGLE_APPLICATION_CREDENTIALS, which I'm assuming is the case based on your use of google.cloud storage.

Now granted you could accept "mylog" or "hivelog" as arguments and make it useful in other scenarios. Also for splitting your filename,should you ever need to go that route again, one line will do the trick:

_,year,month,data,filename = data['name'].split('/')

In that scenario, the underscore just serves to tell yourself and others that you don't plan to use that portion of the split.

You could use extended unpacking to ignore multiple values such as

*_,month,day,filename = data['name'].split('/')

Or you could combine the two

*_,month,day,_ = data['name'].split('/')

edit: link to gcsfs documentation

user_130674900
Related