I have a Go web application which is uploading two files to two separate buckets using two separate clients in GCP using a method called UploadFile(). When I call this function synchronously, each function call takes about 200ms (time N).
However, if I execute the exact same function inside of Go routine, each call individually takes 2N. The entire resulting execution time also being doubled. Here's some example code:
var wg sync.WaitGroup
wg.Add(2)
// ASync upload 1
go func() {
UploadObject(fileBytes, app.files.Bucket(app.config.storage.fileBucket), fileID)
wg.Done()
}()
// ASync upload 2
go func() {
UploadObject(fileBytes, app.files.Bucket(app.config.storage.regionalFileBucket, fileID)
wg.Done()
}()
wg.Wait()
Here is the implementation of UploadObject()
func UploadObject(uploadBytes []byte, bucket *storage.BucketHandle, objectName string) error {
// Will cancel request after 30 seconds if upload is not complete.
writeContext, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
// Upload an object with storage.Writer.
objectWriter := bucket.Object(objectName).NewWriter(writeContext)
if _, err := io.Copy(objectWriter, bytes.NewBuffer(uploadBytes)); err != nil {
return fmt.Errorf("io.Copy: %v", err)
}
// Data can continue to be added to the file until the writer is closed.
if err := objectWriter.Close(); err != nil {
return fmt.Errorf("Writer.Close: %v", err)
}
return nil
}
From inspecting the logs, and writing a bunch of timing around calls, I was able to note the correlation of objectWriter.Close() for both routines always being called one after another - as if to suggest they are locking/blocking each other, but I cant imagine how or why that would be the case.
I've also deployed this code to my GCP Cloud Run containers to try and isolate if it was just a localhost issue - again, no luck.
This also happens if only one of the two calls is in a Go routine. For example, if we do something like this, we still get the same 2N outcome for each functions runtime:
var wg sync.WaitGroup
wg.Add(1)
go func() {
UploadObject(fileBytes, app.files.Bucket(app.config.storage.fileBucket), fileID)
wg.Done()
}()
UploadObject(fileBytes, app.files.Bucket(app.config.storage.regionalFileBucket, fileID)
wg.Wait()
From my vantage point, I can't see any relationship between these cloud storage calls during execution, but perhaps maybe there is something about the Go GCP client and concurrency I don't yet understand. Hoping someone can help shed some light.