AWS Systems Manager - Instance not showing

Viewed 6010

Could anyone help me investigate an issue with EC2 instance profile? I have create an EC2 instance and I put an IAM role. But, when I check on the instance I see: No roles attached to instance profile: xxx-instance-profile.

Any idea where I have to look? Because, when I check that instance profile (role), I have this in the trust:

Trusted entities The identity provider(s) ec2.amazonaws.com

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

I have attached one permission policy AmazonSSMManagedInstanceCore

When I go to my instance, I see that no roles are attached. And, in Systems Manager -> Session Manager, I don't see my instances.

I have no clue what I'm doing wrong :( Any suggestions?

Regards,

7 Answers

I am not sure what you mean by an issue with EC2 instance profile. Instance profiles are permission sets that you grant to an EC2 instance, by defining a policy that contains the permissions required and attaching that policy to a role. The role is attached to an EC2 instance. Because the role will be used by a service it must have a trust relationship to that service.

Which Systems Manager service do you want to use? You can create your own custom policy with specific services and restrictions to specific AWS instances. Or you can use the managed policies.

Here are some example of various policies.

Let's suppose you want a role attached to an EC2 instance so that you can remotely login to that instance using Systems Manager Session Manager.

Let's assume the instance is in a VPC that has a route to the internet, either directly via the Internet Gateway or via a NAT Gateway to the Internet Gateway.

In this case, your instance has a route to the AWS Public Service for Systems Manager Session Manager. The instance must have the Systems Manager Session Manager agent installed. This agent is pre-installed on Amazon Linux 2, Amazon Linux and Ubuntu 16.04, 18.04, 20.04.

Assuming the agent is installed and there is a route to the service, then your instance as you mentioned need rights via IAM to access the service. This is done by granting a role to the EC2 instance.

  • To do this go to IAM - https://console.aws.amazon.com/iam/.
  • Select Roles from the navigation panel, create a new role
  • Select Type of trusted entity as AWS Service
  • Choose the EC2 option under Common Use cases
  • Press Next:Permissions
  • Here you can create a custom policy if you want, I suggest using a managed policy
  • Select an existing managed policy by searching for AmazonEC2RoleforSSM, there are other SSM managed policies, AmazonEC2RoleforSSM is specific for the management of EC2
  • select it and press next:tags
  • press next:review,
  • give it a name - my-ec2-ssm-role

Now we have a role for the EC2 instance, next we need to add that role to the instance.

Go to EC2 - https://console.aws.amazon.com/ec2

  • select your instance
  • from the menu on the top right, select actions, security, modify IAM role.
  • select the role you just created my-ec2-ssm-role
  • press save

Now that the role is linked go to Systems Manager Session Manager https://console.aws.amazon.com/systems-manager/session-manager

  • Press Start session
  • Your instance should be visible, and you can select it and press start session

If you instance is not visible, it could be that you do not have a route to the AWS Service Endpoints. For example the EC2 instance is not in a public subnet or does not have a route to the internet. In this case you need to add 3 VPC endpoints to your subnet. These endpoints are:

  • com.amazonaws.[region].ssm
  • com.amazonaws.[region].ssmmessages
  • com.amazonaws.[region].ec2messages

You can read how to set it up here.

After attaching the AmazonSSMManagedInstanceCore policy to an existing EC2 role, I had to reboot the EC2 instance before it showed up in Systems Manager. Thanks to @Jason who mentioned this in a comment.

You can run AWSSupport-TroubleshootManagedInstance runbook to check what it is missing in your instance's configuration.

If you need to make any change after the troubleshoot like adding an IAM role make sure to restart, the ssm agent in the ec2 instance in order to make it visible in the registered managed instances.

Answering "Systems Manager -> Session Manager, I don't see my instances" --Do you see your managed instances in Fleet Manager? One reason why Instances are not visible to the Systems manager is if the instance has no ssm agent installed. Eg: Ubuntu comes with ssm pre-installed but RHEL does not have ssm pre-installed. Check this out : https://aws.amazon.com/premiumsupport/knowledge-center/systems-manager-ec2-instance-not-appear/

Systems manager immediately showed my ubuntu instances, for RHEL instances I had to manually install ssm agent. https://docs.aws.amazon.com/systems-manager/latest/userguide/agent-install-rhel.html This might be the reason why you cant see instances in session manager as well.

I had the same issue with all of my EC2 instances not showing up in Session Manager, even though they had the correct security/networking set up, turns out I had to go to Systems Manager -> Session Manager -> Preferences and Enable KMS encryption.

There are a few scenarios in which ssm can be deployed and break. All of this assumes you have the proper role attached to the vm.

resource aws_iam_role "ssm" {
    name = "myssm"
    assume_role_policy = <<EOF
{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Action": "sts:AssumeRole",
     "Principal": {
       "Service": "ec2.amazonaws.com"
     },
     "Effect": "Allow",
     "Sid": ""
   }
 ]
}
EOF
}

resource aws_iam_role_policy_attachment "ssm" {
    policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    role = aws_iam_role.ssm.id
}

resource aws_iam_instance_profile "ssm" {
    name = "myssm"
    role = aws_iam_role.ssm.id
}

resource "aws_instance" "private" {
  ami                    = data.aws_ami.amzn2.id
  instance_type          = "t3.micro"
  subnet_id              = aws_subnet.private.id
  vpc_security_group_ids = [aws_security_group.test-ssm.id]

  iam_instance_profile = aws_iam_instance_profile.ssm.name


  tags = {
      Name = "session-manager-private"
  }
}

public subnet with no public ip (internet access) no public ip

In this scenario even though you have a vm in a public subnet and outbound via the igw, there's no public ip on the vm so ssm will not work.

public subnet with public ip (internet access) with public ip

In this scenario, same as the previous only difference being I've added a public ip to the vm and ssm kicks into life.

private subnet with public ip (internet access) enter image description here

In this scenario even though the vm is in a private subnet it has outbound internet access via a public nat gateway which in turn has outbound access via the internet gateway. The nat gateway picutred here has a public ip. When a nat gw with a public ip sits infront of a private subnet those vms use that pubic ip for internet outbound, so ssm works.

enter image description here

Well I wonder what's going to happen here? If you guessed absolutely nothing, you'd be right. There's no public ip no route out of any kind and no way in. It's a completely isolated tenant. So how do you get ssm working this scenario? Next diagram.

private subnet with no public ip (no internet access) enter image description here

In this instance, you need to add vpc endpoints - unsurprisingly to the vpc - and then associate them with the private subnet you want to connect into. Endpoints are created at vpc level and then "associated". The ssm endpoints are of type "interface" so an eni is created in that subnet for each endpoint and a private dns zone is set up so that the vm sends traffic to the local ssm enis and not to the aws fabric globally.

Here's some terraform to do it, sg is allowing 443.

locals {
    endpoints= toset([
        "com.amazonaws.eu-west-2.ssm",
        "com.amazonaws.eu-west-2.ssmmessages",
        "com.amazonaws.eu-west-2.ec2messages"
    ])
}

resource aws_vpc_endpoint "endpoints" {
    for_each          = local.endpoints
    vpc_id            = data.aws_vpc.main.id
    service_name      = each.key
    vpc_endpoint_type = "Interface"

    security_group_ids = [
        aws_security_group.test-ssm.id
    ]

  private_dns_enabled = true
}

resource "aws_vpc_endpoint_subnet_association" "association" {
  for_each        = local.endpoints
  vpc_endpoint_id = aws_vpc_endpoint.endpoints[each.key].id
  subnet_id       = aws_subnet.completely-private.id
}

In my case, it took about 30 minutes for EC2 instance to appear in Fleet Manager. I had existing EC2 without any attached IAM service role. I created a new IAM role with AmazonSSMManagedInstanceCore and AmazonEC2RoleforSSM permission policies and in about 30 minutes my EC2 popped up in Fleet Manager. BTW, Windows platform EC2 instance also comes with preinstalled SSM Agent.

Related