Ansible search a csv variable in a json query

Viewed 65

I just started working with ansible and actually, my purpose for this topic is to define a dynamic CSV file variables , search in a json query and execute some "PUT" "PATCH" etc. operation in RESTAPI, which have emerged items in search. Let give examples;


host-list.csv

serialnumber
2J201024GR
2J201012GR
2J201032GR
2J201008GR

list.yml

tasks:

 - read_csv:
    path: /home/ansible/host_list.csv
    key: serialnumber
   register: vms
   delegate_to: localhost

 - name: List Of Task
   include_tasks: 03-search-profile.yml
   loop: " {{ vms.dict|dict2items }}"

search.yml

  - name: List Of Hardware Info
    no_log: True
    set_fact:
      server_name: "{{ item.name }}"
      SN: "{{ item.serialNumber }}"
      profileuri: "{{ item.serverProfileUri }}"
      serverHardwareUri : "{{ item.uri }}"
    with_items: "{{ server_hardware_result.json.members | json_query('[*]') }}"
    when: "serial_nbr in item.serialNumber"

  - name: Collection of Variables
    debug:
      msg: The Name is {{ server_name }}, The serialnumber is {{ SN }} ,  Serverhardware uri is {{ serverHardwareUri }}  and serverprofileuri is {{ profileuri }}

with a single defined "serial_nbr" variable, I can perform all tasks, which I wanted. Unfortunately, when I want to loop more than one "serial number" for search and further tasks, I can not do any "block- loop" or "include task - set fact". ( ansible not allowed).

I need the set_fact items in the JSON query for further task execution, and also need to search the given serial numbers in the JSON query. Thank you for your help.

Regards

2 Answers

I assume that search.yml corresponds to your include 03-search-profile.yml?

Since you have already moved the search to a separate yml file, you can include it multiple times, setting the variable serial_nbr differently.

In list.yml you can define the following tasks:

- read_csv:
    path: /home/ansible/host_list.csv
    key: serialnumber
  register: vms
  delegate_to: localhost


- name: List Of Task
  include_tasks: 03-search-profile.yml
  vars:
    serial_nbr: 2J201024GR
  loop: " {{ vms.dict|dict2items }}"

- debug:
    msg: "Found! Server {{ server_name }} for serial {{ SN }}"
  when: server_found


- name: List Of Task
  include_tasks: 03-search-profile.yml
  vars:
    serial_nbr: something_unknown
  loop: " {{ vms.dict|dict2items }}"

- debug:
    msg: "Found! Server {{ server_name }} for serial {{ SN }}"
  when: server_found

- debug:
    msg: "Server NOT found!"
  when: not server_found

The file 03-search-profile.yml (search.yml?) you could extend as follows. I added a variable server_found, which is first set to no, if the serial number is found, it is set to yes. This makes it easier to query later.

- name: Set server_found to no
  set_fact:
    server_found: no

- name: List Of Hardware Info
  no_log: True
  set_fact:
    server_name: "{{ item.name }}"
    SN: "{{ item.serialNumber }}"
    profileuri: "{{ item.serverProfileUri }}"
    serverHardwareUri : "{{ item.uri }}"
    server_found: yes
  with_items: "{{ server_hardware_result.json.members | json_query('[*]') }}"
  when: "serial_nbr in item.serialNumber"

- name: Collection of Variables
  debug:
    msg: >
      The Name is {{ server_name }}, The serialnumber is {{ SN }},
      Serverhardware uri is {{ serverHardwareUri }} and
      serverprofileuri is {{ profileuri }}

I hope this helps you with your problem.


Note: In your debug task you have a very long string with Jinja expressions. If you write this as one line, as you did, you have to use quotes. Alternatively, you can write this using folded notation with >.


Addition:

If you read your CSV file with read_csv you get a dict about the serial numbers, with { vms.dict | list }} you get a list of all your serial numbers you have read.

However, I'm not sure what exactly your goal is, and what you wanted to achieve with your loop. Also, you don't write anything about the variable server_hardware_result.

Assuming that the variable server_hardware_result looks like Vladimir Botka mentioned, here are some examples of how you can work with the data:

  • Data structure of server_hardware_result

    server_hardware_result:
      json:
        members:
          - name: alice
            serialNumber: 2J201024GR
            serverProfileUri: 10.1.0.11/profile/uri
            uri: 10.1.0.11/uri
          - name: bob
            serialNumber: 2J201012GR
            serverProfileUri: 10.1.0.12/profile/uri
            uri: 10.1.0.12/uri
          - name: eve
            serialNumber: 2J201032GR
            serverProfileUri: 10.1.0.13/profile/uri
            uri: 10.1.0.13/uri
          - name: mallory
            serialNumber: 2J201008GR
            serverProfileUri: 10.1.0.14/profile/uri
            uri: 10.1.0.14/uri
    
  • Content of host_list.csv

    serialnumber
    2J201024GR
    2J201012GR
    2J201048GR
    2J201032GR
    2J201008GR
    2J201009GR
    

A set of tasks to show examples in handling the data:

- read_csv:
    path: host_list.csv
    key: serialnumber
  register: vms
  delegate_to: localhost

- name: Data structure of vms
  debug:
    var: vms

- name: Read serials as a list
  debug:
    msg: "All serials from csv file: {{ vms.dict | list }}"

- name: Make dict by serial from server_hardware_result members
  set_fact:
    server_hardware_dict: "{{ dict( keys | zip(members) ) }}"
  vars:
    members: "{{ server_hardware_result.json.members }}"
    keys: "{{ members | map(attribute='serialNumber') }}"

- name: Data structure of server_hardware_dict
  debug:
    var: server_hardware_dict

- name: Check if csv serials are in server_hardware_dict
  debug:
    msg: "Is serial {{ item }} in server_hardware_dict? {{ item in server_hardware_dict }}"
  with_items:
    - "{{ vms.dict | list }}"

- name: Print name of found server_hardware
  debug:
    msg: "{{ server_hardware_dict[item].name }}"
  when: item in server_hardware_dict
  with_items:
    - "{{ vms.dict | list }}"

Results and Explanation

  • Task read_csv should be clear.

    TASK [read_csv] **************************************************************
    ok: [localhost]
    
  • The data structure of vms registered from read_csv

    TASK [Data structure of vms] *************************************************
    ok: [localhost] => {
        "vms": {
            "changed": false,
            "dict": {
                "2J201008GR": {
                    "serialnumber": "2J201008GR"
                },
                "2J201009GR": {
                    "serialnumber": "2J201009GR"
                },
                "2J201012GR": {
                    "serialnumber": "2J201012GR"
                },
                "2J201024GR": {
                    "serialnumber": "2J201024GR"
                },
                "2J201032GR": {
                    "serialnumber": "2J201032GR"
                },
                "2J201048GR": {
                    "serialnumber": "2J201048GR"
                }
            },
            "failed": false,
            "list": []
        }
    }
    
  • {{ vms.dict | list }} delivers you a list of all keys from the dict.

    TASK [Read serials as a list] ************************************************
    ok: [localhost] => {
        "msg": "All serials from csv file: ['2J201024GR', '2J201012GR', '2J201048GR', '2J201032GR', '2J201008GR', '2J201009GR']"
    }
    
  • server_hardware_result.json.members is a list, but you can transform it into a dict, keyed by serialNumber.

    TASK [Make dict by serial from server_hardware_result members] ***************
    ok: [localhost]
    
  • After creating the server_hardware_dict, the whole thing looks like follows. Now you can simple work with it by using the serial number.

    TASK [Data structure of server_hardware_dict] ********************************
    ok: [localhost] => {
        "server_hardware_dict": {
            "2J201008GR": {
                "name": "mallory",
                "serialNumber": "2J201008GR",
                "serverProfileUri": "10.1.0.14/profile/uri",
                "uri": "10.1.0.14/uri"
            },
            "2J201012GR": {
                "name": "bob",
                "serialNumber": "2J201012GR",
                "serverProfileUri": "10.1.0.12/profile/uri",
                "uri": "10.1.0.12/uri"
            },
            "2J201024GR": {
                "name": "alice",
                "serialNumber": "2J201024GR",
                "serverProfileUri": "10.1.0.11/profile/uri",
                "uri": "10.1.0.11/uri"
            },
            "2J201032GR": {
                "name": "eve",
                "serialNumber": "2J201032GR",
                "serverProfileUri": "10.1.0.13/profile/uri",
                "uri": "10.1.0.13/uri"
            }
        }
    }
    
  • {{ item in server_hardware_dict }} returns True or False, depending on whether the serial number is in server_hardware_dict or not.

    TASK [Check if csv serials are in server_hardware_dict] **********************
    ok: [localhost] => (item=2J201024GR) => {
        "msg": "Is serial 2J201024GR in server_hardware_dict? True"
    }
    ok: [localhost] => (item=2J201012GR) => {
        "msg": "Is serial 2J201012GR in server_hardware_dict? True"
    }
    ok: [localhost] => (item=2J201048GR) => {
        "msg": "Is serial 2J201048GR in server_hardware_dict? False"
    }
    ok: [localhost] => (item=2J201032GR) => {
        "msg": "Is serial 2J201032GR in server_hardware_dict? True"
    }
    ok: [localhost] => (item=2J201008GR) => {
        "msg": "Is serial 2J201008GR in server_hardware_dict? True"
    }
    ok: [localhost] => (item=2J201009GR) => {
        "msg": "Is serial 2J201009GR in server_hardware_dict? False"
    }
    
  • You can print out name of all serials, read from csv, but skip that one, which are not in server_hardware_dict, work with with_items and when.

    TASK [Print name of found server_hardware] ***********************************
    ok: [localhost] => (item=2J201024GR) => {
        "msg": "alice"
    }
    ok: [localhost] => (item=2J201012GR) => {
        "msg": "bob"
    }
    skipping: [localhost] => (item=2J201048GR)
    ok: [localhost] => (item=2J201032GR) => {
        "msg": "eve"
    }
    ok: [localhost] => (item=2J201008GR) => {
        "msg": "mallory"
    }
    skipping: [localhost] => (item=2J201009GR)
    

I hope that my examples will help you.

Given the file (abridged for testing)

shell> cat host-list.csv
serialnumber
2J201024GR
2J201012GR

simplify the code and put the below declaration into the vars (keep the read_csv task if you have to)

  vms: "{{ lookup('file', 'host-list.csv').splitlines()[1:] }}"

gives

  vms:
  - 2J201024GR
  - 2J201012GR

Let's assume the dictionary server_hardware_result for testing

    server_hardware_result:
      json:
        members:
          - name: alice
            serialNumber: 2J201024GR
            serverProfileUri: 10.1.0.11/profile/uri
            uri: 10.1.0.11/uri
          - name: bob
            serialNumber: 2J201012GR
            serverProfileUri: 10.1.0.12/profile/uri
            uri: 10.1.0.12/uri
          - name: eve
            serialNumber: 2J201032GR
            serverProfileUri: 10.1.0.13/profile/uri
            uri: 10.1.0.13/uri
          - name: mallory
            serialNumber: 2J201008GR
            serverProfileUri: 10.1.0.14/profile/uri
            uri: 10.1.0.14/uri

put the below declarations into the vars

  members_selected: "{{ server_hardware_result.json.members|
                        selectattr('serialNumber', 'in', vms) }}"
  members: "{{ dict(vms|zip(members_selected)) }}"

give the dictionary

  members:
    2J201012GR:
      name: bob
      serialNumber: 2J201012GR
      serverProfileUri: 10.1.0.12/profile/uri
      uri: 10.1.0.12/uri
    2J201024GR:
      name: alice
      serialNumber: 2J201024GR
      serverProfileUri: 10.1.0.11/profile/uri
      uri: 10.1.0.11/uri

Now, use the dictionary depending on your use case. Iterate the items, for example

    - name: Collection of Variables
      debug:
        msg: |-
          server_name: {{ server_name }}
          SN: {{ SN }}
          serverHardwareUri: {{ serverHardwareUri }}
          profileUri: {{ profileUri }}
      loop: "{{ vms }}"
      vars:
        server_name: "{{ members[item].name }}"
        SN: "{{ members[item].serialNumber }}"
        profileUri: "{{ members[item].serverProfileUri }}"
        serverHardwareUri: "{{ members[item].uri }}"

gives (abridged)

TASK [Collection of Variables] ***************************************************************
ok: [localhost] => (item=2J201024GR) => 
  msg: |-
    server_name: alice
    SN: 2J201024GR
    serverHardwareUri: 10.1.0.11/uri
    profileUri: 10.1.0.11/profile/uri
ok: [localhost] => (item=2J201012GR) => 
  msg: |-
    server_name: bob
    SN: 2J201012GR
    serverHardwareUri: 10.1.0.12/uri
    profileUri: 10.1.0.12/profile/uri

Example of a complete playbook for testing

- hosts: localhost

  vars:

    server_hardware_result:
      json:
        members:
          - name: alice
            serialNumber: 2J201024GR
            serverProfileUri: 10.1.0.11/profile/uri
            uri: 10.1.0.11/uri
          - name: bob
            serialNumber: 2J201012GR
            serverProfileUri: 10.1.0.12/profile/uri
            uri: 10.1.0.12/uri
          - name: eve
            serialNumber: 2J201032GR
            serverProfileUri: 10.1.0.13/profile/uri
            uri: 10.1.0.13/uri
          - name: mallory
            serialNumber: 2J201008GR
            serverProfileUri: 10.1.0.14/profile/uri
            uri: 10.1.0.14/uri

    vms: "{{ lookup('file', 'host-list.csv').splitlines()[1:] }}"

    members_selected: "{{ server_hardware_result.json.members|
                          selectattr('serialNumber', 'in', vms) }}"
    members: "{{ dict(vms|zip(members_selected)) }}"

  tasks:

    - debug:
        var: vms
    - debug:
        var: members
    
    - name: Collection of Variables
      debug:
        msg: |-
          server_name: {{ server_name }}
          SN: {{ SN }}
          serverHardwareUri: {{ serverHardwareUri }}
          profileUri: {{ profileUri }}
      loop: "{{ vms }}"
      vars:
        server_name: "{{ members[item].name }}"
        SN: "{{ members[item].serialNumber }}"
        profileUri: "{{ members[item].serverProfileUri }}"
        serverHardwareUri: "{{ members[item].uri }}"
Related