I'm having issues uploading files, that take a bit of time, using presigned put object urls together with transfer acceleration. Sometimes it works and sometimes it doesn't.
I've performed the following tests using Java SDK 2.15.9 to generate urls and curl to upload. Also I'm uploading from Sweden to an s3 bucket located in the us-east-2 region using transfer acceleration.
| file size | url expire time | transfer speed | time to upload | status |
|---|---|---|---|---|
| 20mb | 1min | 100k/s | x | failed after 1min 23sec on 403 forbidden |
| 42mb | 10min | 50k/s | 13min | success |
| 42mb | 10min | 10k/s | x | failed after 12min on 403 forbidden |
| 42mb | 10min | 25k/s | 28min | success |
What is going on here? First theory of mine was that the expire time needed to be longer than the upload time. However reading through the docs it seems that expire time is validated on start of request. Is that true when transfer acceleration is enabled as well? Also using an expire time of 10min worked even tho one upload took 28min.
Do I need to set a longer expire time?
Used the following curl command:
curl -v -H "Content-Type: $contentType" --limit-rate $rateLimit --upload-file $file "$url"
Code to generate URL:
private val presigner by lazy {
S3Presigner.builder()
.credentialsProvider(DefaultCredentialsProvider.create())
.region(s3Region)
.serviceConfiguration(S3Configuration.builder()
.accelerateModeEnabled(true)
.checksumValidationEnabled(false)
.build())
.build()
}
override fun run() {
val url = presigner.presignPutObject { builder ->
builder.putObjectRequest {
it.bucket(s3Bucket)
it.key(UUID.randomUUID().toString())
it.contentType(contentType)
}.signatureDuration(Duration.ofSeconds(expire))
}.url()
println(url)
}