AWS Fargate ResourceInitializationError: unable to pull secrets or registry auth: pull command failed: : signal: killed

user_2800708 0
Viewed 37147

Slightly tearing my hair out with this one... I am trying to run a Docker image on Fargate in a VPC in a Public subnet. When I run this as a Task I get:

ResourceInitializationError: unable to pull secrets or registry auth: pull
command failed: : signal: killed

If I run the Task in a Private subnet, through a NAT, it works. It also works if I run it in a Public subnet of the default VPC.

I have checked through the advice here:

Aws ecs fargate ResourceInitializationError: unable to pull secrets or registry auth

In particular, I have security groups set up to allow all traffic. Also Network ACL set up to allow all traffic. I have even been quite liberal with the IAM permissions, in order to try and eliminate that as a possibility:

The task execution role has:

   {
        "Action": [
            "kms:*",
            "secretsmanager:*",
            "ssm:*",
            "s3:*",
            "ecr:*",
            "ecs:*",
            "ec2:*"
        ],
        "Resource": "*",
        "Effect": "Allow"
    }

With trust relationship to allow ecs-tasks to assume this role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The security group is:

sg-093e79ca793d923ab All traffic All traffic All 0.0.0.0/0

And the Network ACL is:

Inbound
Rule number Type Protocol Port range Source Allow/Deny
100 All traffic All All 0.0.0.0/0    Allow
*   All traffic All All 0.0.0.0/0    Deny

Outbound
Rule number Type Protocol Port range Destination Allow/Deny
100 All traffic All All 0.0.0.0/0    Allow
*   All traffic All All 0.0.0.0/0    Deny

I set up flow logs on the subnet, and I can see that traffic is Accept Ok in both directions.

I do not have any Interface Endpoints set up to reach AWS services without going through the Internet Gateway.

I also have Public IP address assigned to the Fargate instance upon creation.

This should work, since the Public subnet should have access to all needed services through the Internet Gateway. It also works in the default VPC or a Private subnet.

Can anyone suggest what else I should check to debug this?

amazon-ecr aws-fargate amazon-ecs amazon-web-services
8 Answers

One of the potential problems for ResourceInitializationError: unable to pull secrets or registry auth: pull command failed: : signal: killed is disabled Auto-assign public IP. After I enabled it (recreating service from the scrath), task run properly without issues.

enter image description here

user_27438840

For those unlucky souls, there is one more thing to check.

I already had an internet gateway in my VPC, DNS was enabled for that VPC, all containers were getting public IPs and the execution role already had access to ECR. But even so, I was still getting the same error.

Turns out the problem was about Routing Table. The routing table of my VPC didn't include a route for directing outbound traffic to internet gateway so my subnet had no internet access.

Adding the second line to the table that routes 0.0.0.0/0 traffic to internet gateway solved the issue.

enter image description here

user_3516710

I was facing the same issue. But in my case, I was triggering the Fargate Container from the Lambda function using the RunTask operation. So In the RunTask operation, I was not passing the below parameter:

assignPublicIp: ENABLED

After adding this, Container was triggering without any issues.

user_135759900

Edited answer based of feedback from @nathan and @howard-swope

checklist:

  • The VPC has "DNS hostnames" and "DNS resolution" enabled
  • "Task execution role" has access to ECR. e.g. has role AmazonECSTaskExecutionRolePolicy

if task is running on a PUBLIC subnet:

  • The subnets have access to internet. i.e. assigning internet gateway to the subnets.

  • Enable "assign public IP" when creating the task.

if task is running on a PRIVATE subnet:

  • The subnets have access to internet. i.e. assigning NAT gateway to the subnets. ... NAT gateway resides on a public subnet
user_28910270

It turns out that I did not have DNS support enabled for the VPC. Once this is enabled, it works.

I did not see DNS support explicitly mentioned in any docs for Fargate - I guess its pretty obvious or how else will it look up the various AWS services it needs. But thought it worth noting in an answer against this error message.

user_28007080

For AWS Batch using Fargate, this error was triggered by the 'Assign public IP' setting being disabled.

This setting is configurable during Job Definition step. However, it is not configurable in the UI after the Job Definition had already been created.

enter image description here

user_57395140

AWS container runner needs to access to the container repositories, and AWS service.

If you're on a public subnet, the easiest is to "Auto-assign public IP" to have your containers access to internet, even if your app do not need egress access to internet.

Otherwise, if you're using only AWS services (ECR, and no images pulled from docker.io), then you could use VPC endpoints to access ECR/S3/Cloudwatch, and enabling DNS options on your VPC.

For private subnet, it's the same.

If you're using docker.io images, then you need egress access to internet in your subnet anyway.

user_15321750

In my case of dealing with the above error, while running the run-task command(yes, not via Service route), I was not specifying the security group in the aws ecs run-task --network-configuration. This was resulting in the default SG being picked up from the task VPC. My default SG in that VPC had no inbound/outbound rules defined. I added ONLY the outbound rule to allow all traffic to everywhere and the error went away.

My setup is that the ECS/Fargate task will run in a private subnet with ECR connectivity via VPC Interface endpoints. I had the checklist, mentioned above, checked and in addition added the SG rule.

user_26222630
Related