Iterate through nested dicts, retaining key names, in Ansible

user_147202 0
Viewed 50

I have a data structure that looks like this:

all_vms:
  clusterA:
    group1:
      - vm-01
      - vm-02
    group2:
      - vm-03
      - vm-04
  clusterB:
    group1:
      - vm-05
      - vm-06

The key names are not known beforehand. There may be shared "group" key names between clusters.

I want to loop through that data structure and run a task with each of the arrays, but I need the names of the keys they're contained within. The looping task would look something like:

- task:
     vms: "{{ item.value }}"
     group: "{{ item.key }}"
     cluster: "{{ item.parentkey }}"
  loop: "{{ all_vms | ??? }}"

And that would unroll to:

- task:
    vms:
      - vm-01
      - vm-02
    group: group1
    cluster: clusterA
- task:
    vms:
      - vm-03
      - vm-04
    group: group2
    cluster: clusterA
- task:
    vms:
      - vm-05
      - vm-06
    group: group3
    cluster: clusterB

I cannot change the main cluster/group structure, but I can change the structure of the elements that are currently arrays. I have considered just duplicating the keys as values, like this:

all_vms:
  clusterA:
    group1:
      cluster: "clusterA"
      group: "group1"
      vms:
        - vm-01
        - vm-02
    group2:
      cluster: "clusterA"
      group: "group2"
      vms:
        - vm-03
        - vm-04
  clusterB:
    group1:
      cluster: "clusterB"
      group: "group1"
      vms:
        - vm-05
        - vm-06

I would rather not do that, because it's terrible, but I can. But I can't even figure out a way to pop each of those things out into an array. (Edit: Actually, I think figured that out right after posting: all_vms | json_query('*.* | []'). I guess I can go with that if there's not a way to use the tidier data structure.)

Or if I could just use a @!#$% nested loop, if ansible would let me:

- block:
  - task:
      vms: "{{ item.value }}"
      group: "{{ item.key }}"
      cluster: "{{ cluster.key }}"
    loop: "{{ cluster.value | dict2items }}"
  loop: "{{ all_vms | dict2items }}"
  loop_control:
    loop_var: cluster

(Yes, I could do this with include_tasks, but having to have a separate file for a nested loop is just ridiculous.)

Any ideas how to iterate over this data structure without having to resort to a separate file just to do nested looping?

ansible nested-loops data-structures nested loops
2 Answers

Although this is definitely possible using several combinations of filters directly in Ansible/jinja2, my 2 cent: save you nerves and time using a custom filter.

Demo below is for a a filter in the dedicated filters_plugin folder adjacent to your playbook. See the roles and collections documentations for better ways to distribute your filter if need be. Note that this was written real quick to put you on track and although it is fully functional with your above data example, you will need to add some data and errors checks if you intend to use this for a larger audience.

filters_plugins/my_cluster_filters.py

def all_vms_list(vm_dict):
    """Return a list of dicts for all vms with their cluster an group information"""
    vms_list = []
    for cluster, groups in vm_dict.items():
        for group_name, hosts in groups.items():
            for host in hosts:
                current_vm = {
                    'cluster': cluster,
                    'group': group_name,
                    'host': host
                }
                vms_list.append(current_vm)

    return vms_list


class FilterModule(object):
    """my cluster data filters."""

    def filters(self):
        """Return the filter list."""
        return {
            'all_vms_list': all_vms_list
        }

Then the test playbook.yml:

---
- hosts: localhost
  gather_facts: false

  vars:
    all_vms:
      clusterA:
        group1:
          - vm-01
          - vm-02
        group2:
          - vm-03
          - vm-04
      clusterB:
        group1:
          - vm-05
          - vm-06

  tasks:
    - debug:
        msg: "Host {{ item.host }} belongs to group {{ item.group }} inside cluster {{ item.cluster }}"
      loop: "{{ all_vms | all_vms_list }}"

gives:

PLAY [localhost] **************************************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={'cluster': 'clusterA', 'group': 'group1', 'host': 'vm-01'}) => {
    "msg": "Host vm-01 belongs to group group1 inside cluster clusterA"
}
ok: [localhost] => (item={'cluster': 'clusterA', 'group': 'group1', 'host': 'vm-02'}) => {
    "msg": "Host vm-02 belongs to group group1 inside cluster clusterA"
}
ok: [localhost] => (item={'cluster': 'clusterA', 'group': 'group2', 'host': 'vm-03'}) => {
    "msg": "Host vm-03 belongs to group group2 inside cluster clusterA"
}
ok: [localhost] => (item={'cluster': 'clusterA', 'group': 'group2', 'host': 'vm-04'}) => {
    "msg": "Host vm-04 belongs to group group2 inside cluster clusterA"
}
ok: [localhost] => (item={'cluster': 'clusterB', 'group': 'group1', 'host': 'vm-05'}) => {
    "msg": "Host vm-05 belongs to group group1 inside cluster clusterB"
}
ok: [localhost] => (item={'cluster': 'clusterB', 'group': 'group1', 'host': 'vm-06'}) => {
    "msg": "Host vm-06 belongs to group group1 inside cluster clusterB"
}

PLAY RECAP ********************************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
user_94010960

And here is the solution using several combinations of filters directly in Ansible / Jinja.

It combines the first level keys and values with a zip filter, in order to have a know subelements name — 1 — on which we can then use a subelements.

The second level key / value pair is accessible thanks to a dict2items mapped on the first level values.

The task ends up being

- set_fact:
    tasks: "{{ tasks | default([]) + [_task] }}"
  loop: >-
    {{
      all_vms.keys()
      | zip(all_vms.values() | map('dict2items'))
      | subelements([1])
    }}
  loop_control:
    label: "{{ item.0.0 }} — {{ item.1.key }}"
  vars:
    _task:
      task:
        vms: "{{ item.1.value }}"
        group: "{{ item.1.key }}"
        cluster: "{{ item.0.0 }}"

Given the playbook:

- hosts: localhost
  gather_facts: no

  tasks:
    - set_fact:
        tasks: "{{ tasks | default([]) + [_task] }}"
      loop: >-
        {{
          all_vms.keys()
          | zip(all_vms.values() | map('dict2items'))
          | subelements([1])
        }}
      loop_control:
        label: "{{ item.0.0 }} — {{ item.1.key }}"
      vars:
        _task:
          task:
            vms: "{{ item.1.value }}"
            group: "{{ item.1.key }}"
            cluster: "{{ item.0.0 }}"
        all_vms:
          clusterA:
            group1:
              - vm-01
              - vm-02
            group2:
              - vm-03
              - vm-04
          clusterB:
            group1:
              - vm-05
              - vm-06

    - debug:
        var: tasks

This yields:

TASK [set_fact] ***************************************************************
ok: [localhost] => (item=clusterA — group1)
ok: [localhost] => (item=clusterA — group2)
ok: [localhost] => (item=clusterB — group1)

TASK [debug] ******************************************************************
ok: [localhost] => 
  tasks:
  - task:
      cluster: clusterA
      group: group1
      vms:
      - vm-01
      - vm-02
  - task:
      cluster: clusterA
      group: group2
      vms:
      - vm-03
      - vm-04
  - task:
      cluster: clusterB
      group: group1
      vms:
      - vm-05
      - vm-06
user_21235300
Related