Google Cloud Storage: upload parallel performance inside of Go routines is slow

Viewed 78

I have a Go web application which is uploading two files to two separate buckets using two separate clients in GCP using a method called UploadFile(). When I call this function synchronously, each function call takes about 200ms (time N).

However, if I execute the exact same function inside of Go routine, each call individually takes 2N. The entire resulting execution time also being doubled. Here's some example code:

var wg sync.WaitGroup
wg.Add(2)

// ASync upload 1
go func() {
    UploadObject(fileBytes, app.files.Bucket(app.config.storage.fileBucket), fileID)
    wg.Done()
}()

// ASync upload 2
go func() {
    UploadObject(fileBytes, app.files.Bucket(app.config.storage.regionalFileBucket, fileID) 
    wg.Done()
}()
wg.Wait()

Here is the implementation of UploadObject()

func UploadObject(uploadBytes []byte, bucket *storage.BucketHandle, objectName string) error {

    // Will cancel request after 30 seconds if upload is not complete.
    writeContext, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    // Upload an object with storage.Writer.
    objectWriter := bucket.Object(objectName).NewWriter(writeContext)

    if _, err := io.Copy(objectWriter, bytes.NewBuffer(uploadBytes)); err != nil {
        return fmt.Errorf("io.Copy: %v", err)
    }
    // Data can continue to be added to the file until the writer is closed.
    if err := objectWriter.Close(); err != nil {
        return fmt.Errorf("Writer.Close: %v", err)
    }

    return nil
}

From inspecting the logs, and writing a bunch of timing around calls, I was able to note the correlation of objectWriter.Close() for both routines always being called one after another - as if to suggest they are locking/blocking each other, but I cant imagine how or why that would be the case.

I've also deployed this code to my GCP Cloud Run containers to try and isolate if it was just a localhost issue - again, no luck.

This also happens if only one of the two calls is in a Go routine. For example, if we do something like this, we still get the same 2N outcome for each functions runtime:

var wg sync.WaitGroup
wg.Add(1)

go func() {
    UploadObject(fileBytes, app.files.Bucket(app.config.storage.fileBucket), fileID)
    wg.Done()
}()

    UploadObject(fileBytes, app.files.Bucket(app.config.storage.regionalFileBucket, fileID) 
wg.Wait()

From my vantage point, I can't see any relationship between these cloud storage calls during execution, but perhaps maybe there is something about the Go GCP client and concurrency I don't yet understand. Hoping someone can help shed some light.

0 Answers
Related