Docker compose job in ECS context fails to fetch AWS secret to retrieve image from DockerHub

Question

I'm trying to use docker compose to create and launch a small ECS-Fargate cluster with just 2 containers. One of the container images is located in my private repository at DockerHub. However, command

docker compose --file path-to-docker-compose-yml-file up

launched in ECS context keeps failing with the error message:

QuarkustodoService TaskFailedToStart: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to get registry auth from asm: service call has been retried 1 time(s): failed to fetch secret arn:aws:secretsmanager:us-east-2:1071311304...

However, the ecsTaskExecutionRole of my AWS user-id is indeed equipped with the necessary policies to get the secret and to decrypt my DockerHub credentials stored in AWS secret-manager and kms. I'm using DockerHub user-id and access-token as credentials and verified they are working.

Can anyone help or has ideas how to debug the problem?

The complete cmd line output reads:

docker compose --file path-to-docker-compose.yml up
[+] Running 17/17
 - quarkus-todo                      DeleteComplete                                                                                    205.2s
 - Cluster                           DeleteComplete                                                                                    154.2s
 - Quarkustodo8080TargetGroup        DeleteComplete                                                                                    155.5s
 - CloudMap                          DeleteComplete                                                                                    200.2s
 - LogGroup                          DeleteComplete                                                                                    157.5s
 - DbTaskExecutionRole               DeleteComplete                                                                                    147.6s
 - QuarkustodoTaskExecutionRole      DeleteComplete                                                                                    157.5s
 - DefaultNetwork                    DeleteComplete                                                                                    154.2s
 - Quarkustodo8080Listener           DeleteComplete                                                                                    152.3s
 - DefaultNetworkIngress             DeleteComplete                                                                                     82.4s
 - Default8080Ingress                DeleteComplete                                                                                     81.2s
 - DbTaskDefinition                  DeleteComplete                                                                                    127.4s
 - QuarkustodoTaskDefinition         DeleteComplete                                                                                    137.2s
 - QuarkustodoServiceDiscoveryEntry  DeleteComplete                                                                                    106.1s
 - DbServiceDiscoveryEntry           DeleteComplete                                                                                     96.4s
 - DbService                         DeleteComplete                                                                                     90.2s
 - QuarkustodoService                DeleteComplete                                                                                    100.3s
QuarkustodoService TaskFailedToStart: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to get registry auth from asm: service call has been retried 1 time(s): failed to fetch secret arn:aws:secretsmanager:us-east-2:1071311304...

I'm using:

Docker version 20.10.6 on Win10
aws --version
aws-cli/2.2.9 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
ecs-cli --version
ecs-cli version 1.21.0 (bb0b8f0)

This is my my docker-compose.yml file:

version: "3.8"

x-aws-vpc: "vpc-07cdb7bacc9b8010a"
x-aws-loadbalancer: "exter-Publi-FY7S28M1QL7L"

services:
    quarkus-todo: 
        image: bergemannf/mytoolchain:quarkus-todo-ce
        x-aws-pull_credentials: arn:aws:secretsmanager:us-east-2:1111111111111:secret:DockerHubAPIToken-d9RKLn

        ports:
            - target: 8080
              x-aws-protocol: http

    db:
        image: postgres
        environment:
            POSTGRES_USER: ""
            POSTGRES_PASSWORD: ""
            POSTGRES_DB: ""

This is my ecs-param.yml file:

version: 1
task_definition:
  ecs_network_mode: awsvpc
  task_role_arn: arn:aws:iam::107131130437:role/ECSTaskRole
  task_execution_role:  arn:aws:iam::1111111111:role/ecsTaskExecutionRole
  task_size:
    cpu_limit: 256
    mem_limit: 512
  pid_mode: task
  ipc_mode: task
  services:
    quarkus-todo:
      essential: true
      depends_on:
        - container_name: db
          condition: START

      init_process_enabled: false
      healthcheck:
        test: ["CMD", "curl -f http://localhost"]
        interval: 10
        timeout: 5
        retries: 3
        start_period: 180
      secrets:  
        - value_from: arn:aws:secretsmanager:us-east-2:107131130437:secret:DockerHubAPIToken-d9RKLn
#          name: dev/DockerHubAccessToken
    db:
      essential: false
  efs_volumes:
      - name: postgres-db-efs
        filesystem_id: fs-5473872f
        root_directory: /
        access_point: fsap-11111111
run_params:
  network_configuration:
    awsvpc_configuration:
      subnets: 
        - subnet-0af2d8c8faa7f6b9f
        - subnet-039c3a3061848c2a9
      security_groups: 
        - sg-0d52c217fa0f25cfb
      assign_public_ip: ENABLED

user_3146741 · Answer

First and foremost, you mention the ecs-cli and the ecs-param.yml file but the compose/ecs integration does not leverage those (for the records the ECS CLI has evolved into AWS Copilot and dropped compose compatibility which was implemented leveraging the new docker compose you are using).

Having that said, your compose file is actually looking good. You only need to call out the x-aws-pull_credentials parameter and docker will generate a CFN template with all the required wirings. For example, this compose file is what I am using:

version: "3.8"

services:
    myweb: 
        image: mreferre/nginx-custom-site
        x-aws-pull_credentials: arn:aws:secretsmanager:us-west-2:111111111111:secret:dockerhubAccessToken-xyzwh
        ports: 
          - "80:80"

.... and it works just fine

$ docker compose up      
[+] Running 14/14
 ⠿ downloads                   CreateComplete                                                                                                                                                                             190.3s
 ⠿ Cluster                     CreateComplete                                                                                                                                                                               6.0s
 ⠿ CloudMap                    CreateComplete                                                                                                                                                                              47.0s
 ⠿ DefaultNetwork              CreateComplete                                                                                                                                                                               5.0s
 ⠿ MywebTaskExecutionRole      CreateComplete                                                                                                                                                                              20.1s
 ⠿ MywebTCP80TargetGroup       CreateComplete                                                                                                                                                                               1.0s
 ⠿ LogGroup                    CreateComplete                                                                                                                                                                               2.0s
 ⠿ LoadBalancer                CreateComplete                                                                                                                                                                              92.0s
 ⠿ DefaultNetworkIngress       CreateComplete                                                                                                                                                                               1.0s
 ⠿ Default80Ingress            CreateComplete                                                                                                                                                                               1.0s
 ⠿ MywebTaskDefinition         CreateComplete                                                                                                                                                                               3.0s
 ⠿ MywebServiceDiscoveryEntry  CreateComplete                                                                                                                                                                               2.0s
 ⠿ MywebTCP80Listener          CreateComplete                                                                                                                                                                               2.0s
 ⠿ MywebService                CreateComplete                                                                                                                                                                              76.6s
$

[note the image I am using is not private but the whole process of pulling the image and logging into DH does not error out]

The only thing I can think of is if you have a network setup that doesn't allow you to reach the Secrets Manager endpoint?

user_14521269 · Answer

I could solve this issue by the creating a new ecs secret via the following command:

docker ecs secret create  --username   --password

This command then outputs a AWS secret arn as reference to the newly-created secret. This arn I put at the designated place in the docker-compose.yml Then the docker compose up cmd worked and the ECS cluster was raised successfully. Really cool.

Docker compose job in ECS context fails to fetch AWS secret to retrieve image from DockerHub

2 Answers