I am trying to write in S3 in parallel as shown below but somehow it doesn't work for me. I am just writing file bytes into S3.
Below is my code which tries to write files into S3 concurrently:
var wg sync.WaitGroup
for _, v := range clientFiles {
wg.Add(1)
go func(v ClientMapFile) {
defer wg.Done()
err := WriteToS3(v.FileContent, s3Connection, v.FileName, bucketName, v.FolderName)
if err != nil {
fmt.Println(err)
}
}(v)
}
wg.Wait()
I always get an error after sometime as and I don't see file being created in S3.
MultipartUpload: upload multipart failed
caused by: RequestError: send request failed
Here is my WriteToS3 method
func WriteToS3(bytes []byte, s3Client *awss3.S3Client, s3FileName string, s3Bucket string, s3DirPath string) error {
s3err := s3Client.CreateFileOnS3(bytes, s3FileName, s3Bucket, s3DirPath)
return s3err
}
Here is my CreateFileOnS3 method.
func (s3client *S3Client) CreateFileOnS3(data []byte, s3FileName string, s3Bucket string, s3DirPath string) error {
key := fmt.Sprintf("%s/%s", s3DirPath, s3FileName)
_, s3err := s3client.uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(s3Bucket),
Key: aws.String(key),
ContentType: aws.String(http.DetectContentType(data)),
Body: bytes.NewReader(data),
})
return s3err
}
And my uploader is defined as shown below:
uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
u.PartSize = 10 * 1024 * 1024 // 10 MiB
u.Concurrency = 100
})
Is there anything wrong I am doing here? I am re-using same s3Connection for all my concurrent writes. If I write sequentially one by one then it works fine.
Note:
Here v.FileContent is XML file bytes. And here is my S3Client struct:
type S3Client struct {
region string
localDir string
sess *session.Session
svc *s3.S3
downloader *s3manager.Downloader
uploader *s3manager.Uploader
}