"Long" running uploads fail on 403 forbidden using presigned URLs

Viewed 82

I'm having issues uploading files, that take a bit of time, using presigned put object urls together with transfer acceleration. Sometimes it works and sometimes it doesn't.

I've performed the following tests using Java SDK 2.15.9 to generate urls and curl to upload. Also I'm uploading from Sweden to an s3 bucket located in the us-east-2 region using transfer acceleration.

file size url expire time transfer speed time to upload status
20mb 1min 100k/s x failed after 1min 23sec on 403 forbidden
42mb 10min 50k/s 13min success
42mb 10min 10k/s x failed after 12min on 403 forbidden
42mb 10min 25k/s 28min success

What is going on here? First theory of mine was that the expire time needed to be longer than the upload time. However reading through the docs it seems that expire time is validated on start of request. Is that true when transfer acceleration is enabled as well? Also using an expire time of 10min worked even tho one upload took 28min.

Do I need to set a longer expire time?

Used the following curl command:

curl -v -H "Content-Type: $contentType" --limit-rate $rateLimit --upload-file $file "$url"

Code to generate URL:

    private val presigner by lazy {
        S3Presigner.builder()
                .credentialsProvider(DefaultCredentialsProvider.create())
                .region(s3Region)
                .serviceConfiguration(S3Configuration.builder()
                        .accelerateModeEnabled(true)
                        .checksumValidationEnabled(false)
                        .build())
                .build()
    }

    override fun run() {
        val url = presigner.presignPutObject { builder ->
            builder.putObjectRequest {
                it.bucket(s3Bucket)
                it.key(UUID.randomUUID().toString())
                it.contentType(contentType)
            }.signatureDuration(Duration.ofSeconds(expire))
        }.url()
        println(url)
    }
1 Answers

Got a response from AWS support that Cloudfront does not send the request to S3 until it has received the entire body. So the resolution is to increase the expire time of the upload URL.

Related