gitlab-ci runner takes a long time to download / upload to and from S3 cache (without "ServerAddress")

we have a gitlab runner running in a kubernetes cluster on AWS. the testjob:

test-job:
  image: centos
  stage: build
  cache:
    paths:
     - output
  script:
    - mkdir -p output
    - date +"%Y-%m-%d" > output/date.txt
  tags:
    - docker
  when: always
takes about 4 minutes to run - but only 4s if i remove the "cache" section.

my runner's config.toml looks like this

[[runners]]
  name = "gitlab-runner..."
  url = "https://gitlab...."
  token = "....."
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      AccessKey = "...."
      SecretKey = "...."
      BucketName = "runner-cache-bucket-name"
      BucketLocation = "eu-central-1"
  [runners.docker]
    tls_verify = false
    image = "ruby:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0

i also tried using an instance profile (same config but without access/secret key).

this configuration works ... it just hangs a long time before and after the "scripts" part. the config seems to be pretty straightforward.

here is the runner log:

Running with gitlab-runner 14.5.2
  on gitlab-runner-....
Preparing the "docker" executor                                                 00:02
Using Docker executor with image centos ...
Pulling docker image centos ...
Using docker image .....
Preparing environment                                                           00:00
Running on runner-.... 
Getting source from Git repository                                              00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/test/.git/
Checking out db156a5f as master...
Removing output/
Skipping Git submodules setup

# [ here, it hangs for 2 minutes ] <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Restoring cache                                                                 00:01
Checking cache for default...
Downloading cache.zip from https://.......s3.dualstack.eu-central-1.amazonaws.com/project/69/default 
Successfully extracted cache
Executing "step_script" stage of the job script 00:01
Using docker image ....
$ mkdir -p output
$ date +"%Y-%m-%d" > output/date.txt

# [ here, it hangs again for 2 minutes ] <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Saving cache for successful job                                                 00:01
Creating cache default...
output: found 2 matching files and directories     
Uploading cache.zip to https://.........s3.dualstack.eu-central-1.amazonaws.com/project/69/default 
Created cache
Cleaning up project directory and file based variables                          00:00
Job succeeded

note that the times on the right add up to more or less 4s - the time it takes with caching disabled.

EDIT ===================================

I added this to the configuration:

ServerAddress = "s3.amazonaws.com"

Now it works as expected. So, my question would be: why does it work - albeit slowly - without that configuration?