Ansible & AWS SSM connectivity/plugin & “ciphertext refers to a customer master key that does not exist”

user_839369 0
Viewed 5277

Anyone able to get ansible's: ansible_connection: aws_ssm working?

AFAICT this should be a drop in replacement for ssh: https://docs.ansible.com/ansible/latest/collections/community/aws/aws_ssm_connection.html

My playbook runs with ssh, but not ssm:

---
- name: Test command
  gather_facts: false
  hosts: all
  vars:
    ansible_connection: ssh
#    ansible_connection: aws_ssm   <--- this one no worky
    ansible_aws_ssm_region: eu-central-1

  tasks:
    - name: test
      command:
        cmd: ls -l

Running using:

ansible-playbook -i inventory_aws_ec2.yml --limit nghc-sbox2-bastion test.yml -vvvv

I’m missing something on the ansible SSM config. The error is: (from /var/log/amazon/ssm/amazon-ssm-agent.log)

2021-08-10 23:48:51 INFO [ssm-session-worker] [bruce.edge@xxx.com-04d88576fd5ec3ae7] [DataBackend] [pluginName=Standard_Stream] Initiating Handshake 2021-08-10 23:48:54 ERROR [ssm-session-worker] [bruce.edge@xxx.com-04d88576fd5ec3ae7] [DataBackend] [pluginName=Standard_Stream] Fetching data key failed: Unable to retrieve data key, Error when decrypting data key AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

The ansible output is no more helpful:

<i-0c208bc6d31fa6bf1> EXEC stdout line:
<i-0c208bc6d31fa6bf1> EXEC stdout line: Starting session with SessionId: bruce.edge@xxx.com-0f7b6c9323afa74bc
<i-0c208bc6d31fa6bf1> EXEC remaining: 60
<i-0c208bc6d31fa6bf1> EXEC remaining: 59
<i-0c208bc6d31fa6bf1> EXEC stdout line:
<i-0c208bc6d31fa6bf1> EXEC stdout line:
<i-0c208bc6d31fa6bf1> EXEC stdout line: SessionId: bruce.edge@xxx.com-0f7b6c9323afa74bc :
<i-0c208bc6d31fa6bf1> EXEC stdout line: ----------ERROR-------
<i-0c208bc6d31fa6bf1> EXEC stdout line: Encountered error while initiating handshake. Fetching data key failed: Unable to retrieve data key, Error when decrypting data key AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
<i-0c208bc6d31fa6bf1> EXEC stdout line:         status code: 400, request id: 53549e47-03a1-4a1f-8f30-8f0c27482cc5
<i-0c208bc6d31fa6bf1> EXEC stdout line:
<i-0c208bc6d31fa6bf1> EXEC stdout line:
<i-0c208bc6d31fa6bf1> ssm_retry: attempt: 0, caught exception(local variable 'returncode' referenced before assignment) from cmd (echo ~...), pausing for 0 seconds
<i-0c208bc6d31fa6bf1> CLOSING SSM CONNECTION TO: i-0c208bc6d31fa6bf1
<i-0c208bc6d31fa6bf1> TERMINATE SSM SESSION: bruce.edge@xxx.com-0f7b6c9323afa74bc
<i-0c208bc6d31fa6bf1> ESTABLISH SSM CONNECTION TO: i-0c208bc6d31fa6bf1
<i-0c208bc6d31fa6bf1> SSM COMMAND: ['/usr/local/bin/session-manager-plugin', '{"SessionId": "bruce.edge@xxx.com-0d95f1030d63fa155", "TokenValue": "......Gsoj8bEu3d9s=", "StreamUrl": "wss://ssmmessages.eu-central-1.amazonaws.com/v1/data-channel/bruce.edge@xxx.com-0d95f1030d63fa155?role=publish_subscribe", "ResponseMetadata": {"RequestId": "8d20fbe9-d3d2-44e7-a832-a1d4d86861a9", "HTTPStatusCode": 200, "HTTPHeaders": {"server": "Server", "date": "Wed, 11 Aug 2021 00:43:13 GMT", "content-type": "application/x-amz-json-1.1", "content-length": "651", "connection": "keep-alive", "x-amzn-requestid": "8d20fbe9-d3d2-44e7-a832-a1d4d86861a9"}, "RetryAttempts": 0}}', 'eu-central-1', 'StartSession', '', '{"Target": "i-0c208bc6d31fa6bf1"}', 'https://ssm.eu-central-1.amazonaws.com']
<i-0c208bc6d31fa6bf1> SSM CONNECTION ID: bruce.edge@xxx.com-0d95f1030d63fa155
<i-0c208bc6d31fa6bf1> EXEC echo ~
<i-0c208bc6d31fa6bf1> _wrap_command: 'echo QTPJHrIizAXitS...

My SSM is setup correctly for other functionality. I’m able to ssh over ssm and run remote playbooks via ssm, just not use the: ansible_connection: aws_ssm connection mechanism.

aws-ssm ansible amazon-ec2
2 Answers

Don't disable KMS encryption as some SSM services won't work.

The right solution is to go to Key Management Service (KMS), select Customer managed keys and select the key you are using.

There you can add the role that your EC2 instances are using as users to that key.

EKS Console

user_39269750

Disabling KMS encryption in the SSM config fixes this issue:

(AWS console -> system manager -> session manager -> preferences tab)

enter image description here

  • Addendum re: KMS May be justified and the lack of appropriate error messages is just super-misleading, see AWS docs on this: https://aws.amazon.com/premiumsupport/knowledge-center/ssm-session-manager-failures/ ie: should work with KMS enabled, just requires making the key accessible for both the anisble user and the target(s), which is obvious when you think about it. Just super-non-obvious from the error. Have not tested

And... need to reconfigure dash NOT to be the default:

sudo dpkg-reconfigure dash

or, for ansible fans:

# See "/var/cache/debconf/config.dat" for name of config item after changing manually
- name: aws-ssm ansible plugin fails if dash is the default shell
  ansible.builtin.debconf:
    name: dash/sh
    question: dash/sh
    value: false
    vtype: boolean
user_8393690
Related