NoCredentialProviders error when running "docker compose up" with AWS ECS integration

Viewed 308

I'm getting the following error when I try to run docker compose up to deploy my infrastructure to AWS using Docker's ECS integration. Note that I'm running this on Pop!_OS 21.10, which is based on Ubuntu.

NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Things I've tried, based on an exhaustive search of SO and other sites:

  • Verified the proper format of my ~/.aws/config and ~/.aws/credentials files are formatted correctly, are in the proper place, and have the correct permissions
  • Verified that the aws cli works fine
  • Verify that AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION are all set correctly
  • Tried copying the config and credentials to /root/.aws
  • Tried setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION in the root user's environment
  • Created /etc/systemd/system/docker.service.d/aws-credentials.conf and populated it with:
[Service]
Environment="AWS_ACCESS_KEY_ID=********************"
Environment="AWS_SECRET_ACCESS_KEY=****************************************"
  • Ran docker -l debug compose up (Only extra information it provides is DEBUG deploying on AWS with region="us-east-1"

I'm running out of options. If anyone has any other ideas to try, I'd love to hear it. Thanks!

Update: I've also now tried the following, with no luck:

  • Tried setting Environment="AWS_SHARED_CREDENTIALS_FILE=/home/kespan/.aws/credentials
  • Tried setting Environment="AWS_SHARED_CREDENTIALS_FILE=/home/kespan/.aws/credentials in /etc/systemd/system/docker.service.d/override.conf
  • After remembering my IAM account has MFA enabled, generated a token and added Environment="AWS_SESSION_TOKEN=..." to override.conf

Also to note - each time after I've added/modified files under /etc/systemd/system/docker.service.d/ I've run:

sudo systemctl daemon-reload
sudo systemctl restart docker

Edit:

Here's one of the Dockerfiles (both the scraper and scheduler use an identical Dockerfile):

FROM denoland/deno:alpine
WORKDIR /app
USER deno
COPY deps.ts .
RUN deno cache --unstable --no-check deps.ts
COPY . .
RUN deno cache --unstable --no-check mod.ts
RUN mkdir -p /var/tmp/log
CMD ["run", "--unstable", "--allow-all", "--no-check", "mod.ts"]

Here's my docker-compose (some bits redacted):

version: '3'
services:
  grafana:
    container_name: grafana
    image: grafana/grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana:/var/lib/grafana
    deploy:
      replicas: 1
  scheduler:
    image: scheduler
    x-aws-pull-credentials: "arn..."
    container_name: scheduler
    environment:
      DB_CONNECTION_STRING: "postgres://..."
      SQS_URL: "..."
      SQS_REGION: "us-east-1"
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
    deploy:
      replicas: 1
  scraper:
    image: scraper
    x-aws-pull-credentials: "arn..."
    container_name: scraper
    environment:
      DB_CONNECTION_STRING: "postgres://..."
      SQS_URL: "..."
      SQS_REGION: "us-east-1"
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
    deploy:
      replicas: 1
volumes:
  grafana:
1 Answers

Have you attempted to use the Amazon ECS Local Container Endpoints tool that AWS Labs provides? It allows you to create an override file for you docker-compose configurations, and it will simulate the ECS endpoints and IAM roles you would be using in AWS.

This is done using the local AWS credentials you have on your workstation. More information is available on the AWS Blog.

Related