Loop through nested loops, but a little bit more restrictve

Viewed 150

I am writing an Ansible playbook that checks and downloads RPM packages of certain versions and patch levels.
I want to make the playbook to stop when there are no new files, so I thought I should try it with stat.

For better understanding, this is what I got:

  • list1 (Major Versions of the package)

    • 13.0
    • 13.1
    • 13.2
  • list2 (Patchversions of the supported major versions)

    • 13.0.0
    • 13.0.1
    • 13.0.2
    • ...
    • 13.1.0
    • 13.1.1
    • 13.1.2
    • ...
    • 13.2.0
    • 13.2.1
    • 13.2.2
    • ...

My tasks looks like this:

  - name: check if file already exists
    stat:
      path: /var/www/html/softwarexyz/{{ item[0] }}/{{ date }}/software-xyz-{{ item[1] }}-xyz.0.x86_64.rpm
    with_nested:
      - "{{ list1.stdout_lines[:] }}"
      - "{{ list2.stdout_lines[:] }}"
    register: stat_results

Ansible does the following:

Check if in folder of version 13.0 all items out of list2 are somehow there, same for 13.1 and 13.2. Of course it makes no sense, that ansible looks in folder 13.0 for versions starting with 13.1 or 13.2.

How do I make this more efficient? As it takes alot of time to check, depending on the list size.

2 Answers

Q: Check folder for version 13.0 and all items of list2, same for 13.1 and 13.2.

A1: Put the patchversions into the list2 and use with_nested. For example

shell> cat playbook.yml
- hosts: localhost
  vars:
    list1:
      - 13.0
      - 13.1
      - 13.2
    list2: [0, 1, 2]
  tasks:
    - debug:
        msg:  /{{ item.0 }}/software-{{ item.0 ~ '.' ~ item.1 }}.rpm
      with_nested:
        - "{{ list1 }}"
        - "{{ list2 }}"

gives

shell> ansible-playbook playbook.yml | grep msg\:
  msg: /13.0/software-13.0.0.rpm
  msg: /13.0/software-13.0.1.rpm
  msg: /13.0/software-13.0.2.rpm
  msg: /13.1/software-13.1.0.rpm
  msg: /13.1/software-13.1.1.rpm
  msg: /13.1/software-13.1.2.rpm
  msg: /13.2/software-13.2.0.rpm
  msg: /13.2/software-13.2.1.rpm
  msg: /13.2/software-13.2.2.rpm

A2: If the patchversions may vary among the versions put the list2 into an attribute and use with_subelements. For example, the playbook below gives the same results

- hosts: localhost
  vars:
    list1:
      - ver: 13.0
        list2: [0, 1, 2]
      - ver: 13.1
        list2: [0, 1, 2]
      - ver: 13.2
        list2: [0, 1, 2]
  tasks:
    - debug:
        msg: /{{ item.0.ver }}/software-{{ item.0.ver ~ '.' ~ item.1 }}.rpm
      with_subelements:
        - "{{ list1 }}"
        - list2

Note: See Migrating from with_X to loop and optionally change with_nested to

      loop: "{{ list1|product(list2)|list }}"

and with_subelements to

      loop: "{{ list1|subelements('list2')|list }}"

You can actually do this from the single list of patch versions. You can adapt the following to keep your two lists if you really need to keep them for whatever reason.

In the below solution, I simply extract the major versions from the list of patch versions, create a consolidated list of patch versions per major versions and loop over it with the subelements lookup.

---
- name: Major / Patch version demo
  hosts: localhost
  gather_facts: false

  vars:
    available_versions:
      - 13.0.0
      - 13.0.1
      - 13.0.2
      - 13.1.0
      - 13.1.1
      - 13.1.2
      - 13.2.0
      - 13.2.1
      - 13.2.2

  tasks:
    - name: Get a list of major versions
      set_fact:
        major_versions: >-
          {{
            available_versions
            | map('regex_replace', '^(\d*\.\d*)\.\d*$', '\g<1>')
            | unique
            | sort
          }}

    - name: Show major versions
      debug:
        var: major_versions

    - name: Create a consolidated list per major version
      set_fact:
        consolidated_versions: >-
          {{
            consolidated_versions | default([])
            +
            [{'major_version': item, 'patch_versions': available_versions | select('contains', item) | list }]
          }}
      loop: "{{ major_versions }}"

    - name: Show our consolidated list
      debug:
        var: consolidated_versions

    - name: Loop over our data structure for each major version
      debug:
        msg: Major version is {{ item.0.major_version }}. Checking patch version {{ item.1 }}
      loop: "{{ lookup('subelements', consolidated_versions, 'patch_versions') }}"

Which gives:

$ ansible-playbook test.yml 

PLAY [Major / Patch version demo] ******************************************************************************************************************************************************************************************************

TASK [Get a list of major versions] ****************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Show major versions] *************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "major_versions": [
        "13.0",
        "13.1",
        "13.2"
    ]
}

TASK [Create a consolidated list per major version] *******************************************************************************************************************************************************************************************************
ok: [localhost] => (item=13.0)
ok: [localhost] => (item=13.1)
ok: [localhost] => (item=13.2)

TASK [Show our consolidated list] ******************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "consolidated_versions": [
        {
            "major_version": "13.0",
            "patch_versions": [
                "13.0.0",
                "13.0.1",
                "13.0.2"
            ]
        },
        {
            "major_version": "13.1",
            "patch_versions": [
                "13.1.0",
                "13.1.1",
                "13.1.2"
            ]
        },
        {
            "major_version": "13.2",
            "patch_versions": [
                "13.2.0",
                "13.2.1",
                "13.2.2"
            ]
        }
    ]
}

TASK [Loop over our data structure for each major version] *****************************************************************************************************************************************************************************
ok: [localhost] => (item=[{'major_version': '13.0'}, '13.0.0']) => {
    "msg": "Major version is 13.0. Checking patch version 13.0.0"
}
ok: [localhost] => (item=[{'major_version': '13.0'}, '13.0.1']) => {
    "msg": "Major version is 13.0. Checking patch version 13.0.1"
}
ok: [localhost] => (item=[{'major_version': '13.0'}, '13.0.2']) => {
    "msg": "Major version is 13.0. Checking patch version 13.0.2"
}
ok: [localhost] => (item=[{'major_version': '13.1'}, '13.1.0']) => {
    "msg": "Major version is 13.1. Checking patch version 13.1.0"
}
ok: [localhost] => (item=[{'major_version': '13.1'}, '13.1.1']) => {
    "msg": "Major version is 13.1. Checking patch version 13.1.1"
}
ok: [localhost] => (item=[{'major_version': '13.1'}, '13.1.2']) => {
    "msg": "Major version is 13.1. Checking patch version 13.1.2"
}
ok: [localhost] => (item=[{'major_version': '13.2'}, '13.2.0']) => {
    "msg": "Major version is 13.2. Checking patch version 13.2.0"
}
ok: [localhost] => (item=[{'major_version': '13.2'}, '13.2.1']) => {
    "msg": "Major version is 13.2. Checking patch version 13.2.1"
}
ok: [localhost] => (item=[{'major_version': '13.2'}, '13.2.2']) => {
    "msg": "Major version is 13.2. Checking patch version 13.2.2"
}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0 
Related