How do you properly loop through a file for tasks within a block in Ansible?

Viewed 617

This has been a challenge and I am stuck on what else to try. Basically, I have a text file that has an unknown amount of lines. I wish to read through each line and set facts for each line that is then used as variables for a win_domain_user module.

If there is only one line in the text file, then I can accomplish my task with no issues or errors. But, if there is more than one line, I do not know how to start the process again with the next line.

For example:

- name: "Pre-task AD | Get user account info"
  win_shell: |
    Get-Content C:\AD\user_info.txt
  register: 'read_output'

The line of text in the user_info.txt file looks like this:

first-John;last-Doe;display_name-Doe, John (US);title-Engineer;employee_type-Employee;user-john.doe;email-john.doe@company.com;customer-Some_Business;dept_groups-Test_Group

Example of two lines:

first-John;last-Doe;display_name-Doe, John (US);title-Engineer;employee_type-Employee;user-john.doe;email-john.doe@company.com;customer-Some_Business;dept_groups-Test_Group
first-Jane;last-Doe;display_name-Doe, Jane (US);title-Engineer II;employee_type-Employee;user-jane.doe;email-jane.doe@company.com;customer-Some_Business;dept_groups-Test_Group

The set facts looks like:

- set_fact:
    first: "{{ read_output.stdout | regex_search(...) }}"
    last: "{{ read_output.stdout | regex_search(...) }}"
    user: "{{ read_output.stdout | regex_search(...) }}"
    email: "{{ read_output.stdout | regex_search(...) }}"
    customer: "{{ read_output.stdout | regex_search(...) }}"
    title: "{{ read_output.stdout  regex_search(...) }}"
    display_name: "{{ read_output.stdout  regex_search(...) }}"
    employee_type: "{{ read_output.stdout |  regex_search(...) }}"
    dept_groups: "{{ read_output.stdout |  regex_search(...) }}"

The winrm Active Directory related module looks like:

- name: "Active Directory | Add domain user account to domain controller"
  win_domain_user:
    firstname: "{{ first }}"
    surname: "{{ last }}"
    name: "{{ first }}{{ space }}{{ last }}"
    description: "{{ title }}"
    password: "{{ password_var_here}}"
    upn: "{{ user }}{{ domain_fqdn_var_here }}"
    email: "{{ email }}"
    state: 'present'
    path: "{{ ou_path_here }}"
    groups: "Domain Users, {{ dept_groups }}"
    attributes:
      displayName: "{{ first }}{{ space }}{{ last }}"
    domain_username: "{{ domainusername_var_here }}"
    domain_password: "{{ domainpassword_var_here }}"
    domain_server: "{{ dc_var_here }}"
  register: 'add_ad_user'

I tried using a win_shell ForEach (PowerShell) loop to action the lines in C:\AD\user_info.txt but I was not able to figure out how to use set_facts within the ForEach loop and use the win_domain_user module within it. So, instead, I looked into using with_items which from what I understand is now loop. From the info above, I was going to create 2 yml files, one that has the 'Get-Content C:\AD\user_info.txt' / register and then use the include_tasks yml with block for the tasks to run against the data from the Get-Content.

The tricky part is that I do not know how many lines are going to be in that user_info.txt file. Am I on the right track as far as using two files for this or is there a more efficient way?

3 Answers

As already stated in the comments, the real good solution would be to change your file format at source to something that is easy to parse. Being in ansible the best format would indeed be yaml/json. But any other standard would make this much easier (xml, csv...).

Note: I know your current data is actually csv with semicolon delimiters. But it doesn't have a header and values contain the field name that must be extracted on each line

As an other remark: registering the output of a cat command (e.g. Get-Content) from a (win-)shell to get a file content is a bad practice. This is precisely why the fetch and slurp modules exist.

The below is just a workaround to get a dict out of your custom and not really ideal format. It will break as soon as a value does not strictly respect the current inferred schema (i.e. contains a dash, a semicolon, a new line...) and will probably become a real nightmare to maintain every time you encounter an exception. But at least it does what you expect from your example and you can build upon that to adapt it to your exact needs.

Edit: my original solution was using a regex (see history if you want to look at it). But @seshadri_c opened my eyes on a much simpler solution that leads to the same result using only split. I borrowed the idea while keeping the key/value mapping dynamic.

Given the following files/user_data.txt

first-John;last-Doe;display_name-Doe, John (US);title-Engineer;employee_type-Employee;user-john.doe;email-john.doe@company.com;customer-Some_Business;dept_groups-Test_Group
first-Jane;last-Doe;display_name-Doe, Jane (US);title-Engineer II;employee_type-Employee;user-jane.doe;email-jane.doe@company.com;customer-Some_Business;dept_groups-Test_Group

The loop_bad_file.yml playbook

---
- name: Loop on a poorly formated file
  hosts: localhost
  gather_facts: false

  vars:
    # File name on the remote (here relative to the current dir for the example....)
    bad_file_name: files/user_data.txt

    # list of user dicts we get by:
    #
    # 1. spliting each line from the original file (`select` removes the last trainling blank line)
    # So we get ['line1....', 'line2....']
    #
    # 2. split each line on semicolons so we get:
    # [['key1-value1', 'key2-value2', ....], ['key1-value1', 'key2-value2', ....], ...]
    #
    # 3. split each field-value on a dash so we get:
    # [[['key1', 'value1'], ['key2', 'value2'], ..., [['key1', 'value1'], ['key2', 'value2'], ...], ...]
    #
    # 4. Turn this back into a list of dicts using the key/values pairs in each element
    user_list: "{{
        (bad_file.content | b64decode).split('\n') | select
        | map('split', ';')
        | map('map', 'split', '-')
        | map('items2dict', key_name=0, value_name=1)
    }}"

  tasks:

    - name: Get file content from remote
      slurp:
        src: "{{ bad_file_name }}"
      register: bad_file

    - name: Show the result using the transformed list of dicts
      vars:
        debug_msg: |-
          This is entry for {{ item.display_name }}
          first: {{ item.first }}
          last: {{ item.last }}
          title: {{ item.title }}
          employee type: {{ item.employee_type }}
          email: {{ item.email }}
          customer: {{ item.customer }}
          department group: {{ item.dept_groups }}
      debug:
        msg: "{{ debug_msg.split('\n') }}"
      loop: "{{ user_list }}"

Gives:

$ ansible-playbook loop_bad_file.yml 

PLAY [Loop on a poorly formated file] *********************************************************************

TASK [Get file content from remote] *********************************************************************
ok: [localhost]

TASK [Show the result using the transformed list of dicts] *********************************************************************
ok: [localhost] => (item={'first': 'John', 'last': 'Doe', 'display_name': 'Doe, John (US)', 'title': 'Engineer', 'employee_type': 'Employee', 'user': 'john.doe', 'email': 'john.doe@company.com', 'customer': 'Some_Business', 'dept_groups': 'Test_Group'}) => {
    "msg": [
        "This is entry for Doe, John (US)",
        "first: John",
        "last: Doe",
        "title: Engineer",
        "employee type: Employee",
        "email: john.doe@company.com",
        "customer: Some_Business",
        "department group: Test_Group"
    ]
}
ok: [localhost] => (item={'first': 'Jane', 'last': 'Doe', 'display_name': 'Doe, Jane (US)', 'title': 'Engineer II', 'employee_type': 'Employee', 'user': 'jane.doe', 'email': 'jane.doe@company.com', 'customer': 'Some_Business', 'dept_groups': 'Test_Group'}) => {
    "msg": [
        "This is entry for Doe, Jane (US)",
        "first: Jane",
        "last: Doe",
        "title: Engineer II",
        "employee type: Employee",
        "email: jane.doe@company.com",
        "customer: Some_Business",
        "department group: Test_Group"
    ]
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Each line from the file will be in the registered variable read_output.stdout_lines. We can loop with this variable and split the line using semi-colon (;) to get the individual attributes of the user.

If we take only the first line, the example below with return John:

"{{ read_output.stdout_lines[0].split(';')[0].split('-')[1] }}"

Similarly a debug task:

- name: "Pre-task AD | Get user account info"
  win_shell: |
    Get-Content C:\AD\user_info.txt
  register: read_output

- name: Show user details after splitting
  debug:
    msg:
      - "First: {{ first }}"
      - "Last: {{ last }}"
      - "Display name: {{ display_name }}"
  vars:
    first: "{{ item.split(';')[0].split('-')[1] }}"
    last: "{{ item.split(';')[1].split('-')[1] }}"
    display_name: "{{ item.split(';')[2].split('-')[1] }}"
  loop: "{{ read_output.stdout_lines }}"

will produce:

    "msg": [
        "First: John",
        "Last: Doe",
        "Display name: Doe, John (US)"
    ]

Continuing with this theme, we can have the win_domain_user task like (not tested):

- name: "Active Directory | Add domain user account to domain controller"
  win_domain_user:
    firstname: "{{ first }}"
    surname: "{{ last }}"
    name: "{{ first }} {{ last }}"
    description: "{{ title }}"
    password: "{{ password_var_here }}"
    upn: "{{ user }}{{ domain_fqdn_var_here }}"
    email: "{{ email }}"
    state: 'present'
    path: "{{ ou_path_here }}"
    groups: "Domain Users, {{ dept_groups }}"
    attributes:
      displayName: "{{ first }} {{ last }}"
    domain_username: "{{ domainusername_var_here }}"
    domain_password: "{{ domainpassword_var_here }}"
    domain_server: "{{ dc_var_here }}"
  vars:
    first: "{{ item.split(';')[0].split('-')[1] }}"
    last: "{{ item.split(';')[1].split('-')[1] }}"
    display_name: "{{ item.split(';')[2].split('-')[1] }}"
    name: "{{ first }} {{ last }}"
    title: "{{ item.split(';')[3].split('-')[1] }}"
    email: "{{ item.split(';')[6].split('-')[1] }}"
    user: "{{ item.split(';')[5].split('-')[1] }}"
    dept_groups: "{{ item.split(';')[8].split('-')[1] }}"
  loop: "{{ read_output.stdout_lines }}"
Related