MultipartUpload error while writing bytes as S3 file

Viewed 205

I am trying to write in S3 in parallel as shown below but somehow it doesn't work for me. I am just writing file bytes into S3.

Below is my code which tries to write files into S3 concurrently:

var wg sync.WaitGroup
for _, v := range clientFiles {
  wg.Add(1)
  go func(v ClientMapFile) {
    defer wg.Done()
    err := WriteToS3(v.FileContent, s3Connection, v.FileName, bucketName, v.FolderName)
    if err != nil {
      fmt.Println(err)
    }      
  }(v)
}
wg.Wait()

I always get an error after sometime as and I don't see file being created in S3.

MultipartUpload: upload multipart failed
caused by: RequestError: send request failed

Here is my WriteToS3 method

func WriteToS3(bytes []byte, s3Client *awss3.S3Client, s3FileName string, s3Bucket string, s3DirPath string) error {
    s3err := s3Client.CreateFileOnS3(bytes, s3FileName, s3Bucket, s3DirPath)
    return s3err
}

Here is my CreateFileOnS3 method.

func (s3client *S3Client) CreateFileOnS3(data []byte, s3FileName string, s3Bucket string, s3DirPath string) error {
    key := fmt.Sprintf("%s/%s", s3DirPath, s3FileName)
    _, s3err := s3client.uploader.Upload(&s3manager.UploadInput{
        Bucket:      aws.String(s3Bucket),
        Key:         aws.String(key),
        ContentType: aws.String(http.DetectContentType(data)),
        Body:        bytes.NewReader(data),
    })
    return s3err
}

And my uploader is defined as shown below:

uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
  u.PartSize = 10 * 1024 * 1024 // 10 MiB
  u.Concurrency = 100
})

Is there anything wrong I am doing here? I am re-using same s3Connection for all my concurrent writes. If I write sequentially one by one then it works fine.

Note:

Here v.FileContent is XML file bytes. And here is my S3Client struct:

type S3Client struct {
    region     string
    localDir   string
    sess       *session.Session
    svc        *s3.S3
    downloader *s3manager.Downloader
    uploader   *s3manager.Uploader
}
0 Answers
Related