Combining arrays in a JSON Object in Ansible

user_2705374 0
Viewed 527

I would like to combine two JSON objects from the two commands below, in order to replace existing S3 notifications with a newer one, using put-bucket-notification-configuration. The combine filter I'm using is currently giving me an error saying that it expects dictionaries.

---
- name: Get existing S3 notifications
  command: >
    aws s3api get-bucket-notification-configuration
      --bucket "{{ s3_notification.bucket }}"
      --region "{{ aws_region }}"
  register: existing_s3_notifications
  environment:
    AWS_ACCESS_KEY_ID: "{{ assumed_role.sts_creds.access_key }}"
    AWS_SECRET_ACCESS_KEY: "{{ assumed_role.sts_creds.secret_key }}"
    AWS_SECURITY_TOKEN: "{{ assumed_role.sts_creds.session_token }}"

- name: Combine existing S3 notifications with new notifications
      command: >
        aws s3api put-bucket-notification-configuration
          --bucket "{{ s3_notification.bucket }}"
          --notification-configuration '{{ s3_notification.configuration | to_json | combine(existing_s3_notifications) }}'
          --region "{{ aws_region }}"
      environment:
        AWS_ACCESS_KEY_ID: "{{ assumed_role.sts_creds.access_key }}"
        AWS_SECRET_ACCESS_KEY: "{{ assumed_role.sts_creds.secret_key }}"
        AWS_SECURITY_TOKEN: "{{ assumed_role.sts_creds.session_token }}"

The JSON objects look like this:

{
"LambdaFunctionConfigurations": [
    {
        "Id": "dev-datamining-wwang-new_product_list",
        "LambdaFunctionArn": "arn:aws:lambda:us-west-2:434815900134:function:dev-datamining-wwang-lambda-update_product_corpus",
        "Events": [
            "s3:ObjectCreated:*"
        ],
        "Filter": {
            "Key": {
                "FilterRules": [
                    {
                        "Name": "Prefix",
                        "Value": "wwang/product_list/"
                    }
                ]
            }
        }
    }
]
}

With another one like this:

{
"LambdaFunctionConfigurations": [
    {
        "Id": "dev-datamining-alee-new_product_list",
        "LambdaFunctionArn": "arn:aws:lambda:us-west-2:434815900135:function:dev-datamining-alee-lambda-update_product_corpus",
        "Events": [
            "s3:ObjectCreated:*"
        ],
        "Filter": {
            "Key": {
                "FilterRules": [
                    {
                        "Name": "Prefix",
                        "Value": "alee/product_list/"
                    }
                ]
            }
        }
    }
],
"TopicConfigurations": [
    <other stuff>
]
}

I'd like the result to be:

{
"LambdaFunctionConfigurations": [
    {
        "Id": "dev-datamining-wwang-new_product_list",
        "LambdaFunctionArn": "arn:aws:lambda:us-west-2:434815900134:function:dev-datamining-wwang-lambda-update_product_corpus",
        "Events": [
            "s3:ObjectCreated:*"
        ],
        "Filter": {
            "Key": {
                "FilterRules": [
                    {
                        "Name": "Prefix",
                        "Value": "wwang/product_list/"
                    }
                ]
            }
        }
    },
    {
        "Id": "dev-datamining-alee-new_product_list",
        "LambdaFunctionArn": "arn:aws:lambda:us-west-2:434815900135:function:dev-datamining-alee-lambda-update_product_corpus",
        "Events": [
            "s3:ObjectCreated:*"
        ],
        "Filter": {
            "Key": {
                "FilterRules": [
                    {
                        "Name": "Prefix",
                        "Value": "alee/product_list/"
                    }
                ]
            }
        }
    }
],
"TopicConfigurations": [
    <other stuff>
]
}

How would I combine these two objects so that the arrays inside the objects are merged together?

ansible amazon-web-services yaml
1 Answers

You need to investigate the extra parameter that the combine filter is offering you:

The filter also accepts an optional recursive=True parameter to not only override keys in the first hash, but also recurse into nested hashes and merge their keys too

{{ {'a':{'foo':1, 'bar':2}, 'b':2} | combine({'a':{'bar':3, 'baz':4}}, recursive=True) }}

This would result in:

{'a':{'foo':1, 'bar':3, 'baz':4}, 'b':2}

See: https://docs.ansible.com/ansible/2.9/user_guide/playbooks_filters.html#combining-hashes-dictionaries

So in your case, you indeed have to combine you two dictionaries first, then, convert them in JSON:

- name: Combine existing S3 notifications with new notifications
      command: >
        aws s3api put-bucket-notification-configuration
          --bucket "{{ s3_notification.bucket }}"
          --notification-configuration '{{ s3_notification.configuration | combine(existing_s3_notifications, recursive=True) | to_json }}'
          --region "{{ aws_region }}"
      environment:
        AWS_ACCESS_KEY_ID: "{{ assumed_role.sts_creds.access_key }}"
        AWS_SECRET_ACCESS_KEY: "{{ assumed_role.sts_creds.secret_key }}"
        AWS_SECURITY_TOKEN: "{{ assumed_role.sts_creds.session_token }}"

Also mind that, if you are on version 2.10 already, you even have another extra parameter:

list_merge: Is a string, its possible values are replace (default), keep, append, prepend, append_rp or prepend_rp. It modifies the behaviour of combine when the hashes to merge contain arrays/lists.

See: https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#combining-hashes-dictionaries, containing nice examples for each values

user_21235300
Related