`aws ecs execute-command` results in `TargetNotConnectedException` `The execute command failed due to an internal error`

user_1767106 0
Viewed 9528

I am running a Docker image on an ECS cluster to shell into it and run some simple tests. However when I run this:

aws ecs execute-command  \
  --cluster MyEcsCluster \
  --task $ECS_TASK_ARN \
  --container MainContainer \
  --command "/bin/bash" \
  --interactive

I get the error:

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

I can confirm the task + container + agent are all running:

aws ecs describe-tasks \
  --cluster MyEcsCluster \
  --tasks $ECS_TASK_ARN \
  | jq '.'

      "containers": [
        {
          "containerArn": "<redacted>",
          "taskArn": "<redacted>",
          "name": "MainContainer",
          "image": "confluentinc/cp-kafkacat",
          "runtimeId": "<redacted>",
          "lastStatus": "RUNNING",
          "networkBindings": [],
          "networkInterfaces": [
            {
              "attachmentId": "<redacted>",
              "privateIpv4Address": "<redacted>"
            }
          ],
          "healthStatus": "UNKNOWN",
          "managedAgents": [
            {
              "lastStartedAt": "2021-09-20T16:26:44.540000-05:00",
              "name": "ExecuteCommandAgent",
              "lastStatus": "RUNNING"
            }
          ],
          "cpu": "0",
          "memory": "4096"
        }
      ],

I'm defining the ECS Cluster and Task Definition with the CDK Typescript code:

    new Cluster(stack, `MyEcsCluster`, {
        vpc,
        clusterName: `MyEcsCluster`,
    })

    const taskDefinition = new FargateTaskDefinition(stack, TestTaskDefinition`, {
        family: `TestTaskDefinition`,
        cpu: 512,
        memoryLimitMiB: 4096,
    })
    taskDefinition.addContainer("MainContainer", {
        image: ContainerImage.fromRegistry("confluentinc/cp-kafkacat"),
        command: ["tail", "-F", "/dev/null"],
        memoryLimitMiB: 4096,
        // Some internet searches suggested setting this flag. This didn't seem to help.
        readonlyRootFilesystem: false,
    })
aws-fargate amazon-ecs docker amazon-web-services
2 Answers

This utility should be able to figure out what's wrong with your setup. Can you give it a try?

user_31467410

Building on @clay's comment

I was also missing ssmmessages:* permissions.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html#ecs-exec-required-iam-permissions says a policy such as

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel"
            ],
            "Resource": "*"
        }
    ]
}

should be attached to the role used in your "task role" (not for the "task execution role"), although the sole ssmmessages:CreateDataChannel permission does cut it.

The managed policies

arn:aws:iam::aws:policy/AmazonSSMFullAccess
arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
arn:aws:iam::aws:policy/AmazonSSMManagedEC2InstanceDefaultPolicy
arn:aws:iam::aws:policy/AWSCloud9SSMInstanceProfile

all contain the necessary permissions, AWSCloud9SSMInstanceProfile being the most minimalistic.

user_117152590
Related