Ansible loop over glob

Ease your own life and simplify / DRY your variables.

Make your defaults/main.yml looks like this, where you don't have to keep on repeating docker_compose_keycloak_config_cli for every single dictionary key:

docker_compose_keycloak_config_cli:
  keycloak_url: http://keycloak:8080
  user: admin
  password: ~
  client_id: admin-cli
  clientsecret: ~
  envvar:
    keycloak_url: http://keycloak:8080

Then your requirement becomes trivial:

{% for key, value in docker_compose_keycloak_config_cli.envvar.items() %}
{{ key | upper }}={{ value }}
{% endfor %}

Which would give you:

KEYCLOAK_URL=http://keycloak:8080

And even better to reduce duplication, use YAML anchors:

docker_compose_keycloak_config_cli:
  keycloak_url: &keycloak_url http://keycloak:8080
  user: admin
  password: ~
  client_id: admin-cli
  clientsecret: ~
  envvar:
    keycloak_url: *keycloak_url

In order to allow overriding it, recursively combine the overriding dictionary into the existing variable. Then use it as described above.

- set_fact:
    docker_compose_keycloak_config_cli: >-
      {{
        docker_compose_keycloak_config_cli
        | combine(
          docker_compose_keycloak_config_cli_override | default({}),
          recursive=True
        )
      }}

The two tasks:

- set_fact:
    docker_compose_keycloak_config_cli: >-
      {{
        docker_compose_keycloak_config_cli
        | combine(
          docker_compose_keycloak_config_cli_override | default({}),
          recursive=True
        )
      }}
  vars:
    docker_compose_keycloak_config_cli:
      keycloak_url: &keycloak_url http://keycloak:8080
      user: admin
      password: ~
      client_id: admin-cli
      clientsecret: ~
      envvar:
        keycloak_url: *keycloak_url
        other_env: foo
    docker_compose_keycloak_config_cli_override:
      envvar:
        keycloak_url: http://keycloak:80

- debug:
    var: docker_compose_keycloak_config_cli

Would yield:

docker_compose_keycloak_config_cli:
  client_id: admin-cli
  clientsecret: null
  envvar:
    keycloak_url: http://keycloak:80
    other_env: foo
  keycloak_url: http://keycloak:8080
  password: null
  user: admin

Q: "Make the template super short. Edit host_vars or group_vars with only the specific variable."

A: For example, given the tree

shell> tree .
.
├── ansible.cfg
├── hosts
├── host_vars
│   ├── host1
│   │   └── keycloak_config_cli.yml
│   ├── host2
│   │   └── keycloak_config_cli.yml
│   └── host3
│       └── keycloak_config_cli.yml
├── pb.yml
└── roles
    └── keycloak
        ├── defaults
        │   └── main.yml
        ├── tasks
        │   └── main.yml
        └── templates
            └── env-vars-keycloak-config-cli.j2

and the inventory

shell> cat hosts
host1
host2
host3

Put the defaults to defaults/main.yml and declare the variable keycloak_config_cli_host. This variable will serve the purpose of overriding the host-specific attributes if defined

shell> cat roles/keycloak/defaults/main.yml 
keycloak_config_cli:
  url: http://keycloak:8080
  user: admin
  password: ~
  client_id: admin-cli
  clientsecret: ~
  timeout: 90
keycloak_config_cli_host: "keycloak_config_cli_{{ inventory_hostname }}"

For testing, create host_vars to override the defaults

shell> cat host_vars/host1/keycloak_config_cli.yml 
keycloak_config_cli_host1:
  timeout: 10

shell> cat host_vars/host2/keycloak_config_cli.yml 
keycloak_config_cli_host2:
  timeout: 20

shell> cat host_vars/host3/keycloak_config_cli.yml 
keycloak_config_cli_host3:
  timeout: 30

Create the tasks. Combine the dictionaries in the first task, optionally display the variable, and display the result of the template

shell> cat roles/keycloak/tasks/main.yml 
- set_fact:
    keycloak_config_cli: "{{ keycloak_config_cli|
                             combine(lookup('vars',
                                            keycloak_config_cli_host,
                                            default={})) }}"
- debug:
    var: keycloak_config_cli
  when: debug|d(false)|bool
- debug:
    msg: "{{ lookup('template', 'env-vars-keycloak-config-cli.j2') }}"

Create the template. It's not very far from your template. Just iterate items, reject empty values, and fit the keys

shell> cat roles/keycloak/templates/env-vars-keycloak-config-cli.j2 
#templates/env-vars-keycloak-config-cli
#jinja2: lstrip_blocks: "True"
{% for key, value in keycloak_config_cli.items() %}
{% if value -%}
KEYCLOAK_{{ key|regex_replace('_','')|upper }}={{ value }}
{% endif -%}
{% endfor %}

The playbook

shell> cat pb.yml 
- hosts: all
  gather_facts: false
  roles:
    - keycloak

gives

shell> ansible-playbook pb.yml 

PLAY [all] **********************************************************************************

TASK [keycloak : set_fact] ******************************************************************
ok: [host1]
ok: [host2]
ok: [host3]

TASK [keycloak : debug] *********************************************************************
skipping: [host1]
skipping: [host2]
skipping: [host3]

TASK [keycloak : debug] *********************************************************************
ok: [host1] => 
  msg: |-
    #templates/env-vars-keycloak-config-cli
    #jinja2: lstrip_blocks: "True"
    KEYCLOAK_URL=http://keycloak:8080
    KEYCLOAK_USER=admin
    KEYCLOAK_CLIENTID=admin-cli
    KEYCLOAK_TIMEOUT=10
ok: [host2] => 
  msg: |-
    #templates/env-vars-keycloak-config-cli
    #jinja2: lstrip_blocks: "True"
    KEYCLOAK_URL=http://keycloak:8080
    KEYCLOAK_USER=admin
    KEYCLOAK_CLIENTID=admin-cli
    KEYCLOAK_TIMEOUT=20
ok: [host3] => 
  msg: |-
    #templates/env-vars-keycloak-config-cli
    #jinja2: lstrip_blocks: "True"
    KEYCLOAK_URL=http://keycloak:8080
    KEYCLOAK_USER=admin
    KEYCLOAK_CLIENTID=admin-cli
    KEYCLOAK_TIMEOUT=30

PLAY RECAP **********************************************************************************
host1: ok=2    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
host2: ok=2    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
host3: ok=2    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0 

shell> cat roles/keycloak/tasks/main.yml
- set_fact:
    keycloak_config_cli: "{{ keycloak_config_cli|
                             combine(lookup('vars',
                                            keycloak_config_cli_group,
                                            default={})) }}"
  loop: "{{ group_names|sort }}"
  vars:
    keycloak_config_cli_group: "keycloak_config_cli_{{ item }}"
  when: enable_group_vars|d(true)|bool

- set_fact:
    keycloak_config_cli: "{{ keycloak_config_cli|
                             combine(lookup('vars',
                                            keycloak_config_cli_host,
                                            default={})) }}"
- debug:
    var: keycloak_config_cli
  when: debug|d(false)|bool
- debug:
    msg: "{{ lookup('template', 'env-vars-keycloak-config-cli.j2') }}"

shell> cat hosts
host1
host2
host3

[devel]
host4

shell> cat group_vars/devel/keycloak_config_cli.yml
keycloak_config_cli_devel:
  timeout: 40

...
ok: [host4] => 
  msg: |-
    #templates/env-vars-keycloak-config-cli
    #jinja2: lstrip_blocks: "True"
    KEYCLOAK_URL=http://keycloak:8080
    KEYCLOAK_USER=admin
    KEYCLOAK_CLIENTID=admin-cli
    KEYCLOAK_TIMEOUT=40

I am very close to a solution, but I can't figure out how to parse the last parts correctly.

  - name: Create list of envvars
    set_fact:
      list_of_envvars: "{{ lookup('ansible.builtin.varnames', 'docker_compose_keycloak_config_cli_envvar_.+$').split(',') }}"

#jinja2 template file
{% for envvar in list_of_envvars %}
{{ vars[envvar] }}
{% endfor %}

If I try to loop over the vars[envvar] it will only give gime the key of the dict and if I just print it out, it will give me the whole dict.

trying to use lookup('dict', envvar) gives an error. So the only way was to lookup('vars', envvar) which gave me the right thing, but I cant seem to parse it correctly.

The output to the file

+{'KEYCLOAK_URL': 'http://keycloak:8080'}
 KEYCLOAK_URL
 {'KEYCLOAK_URL': 'http://keycloak:8080'}
+{'KEYCLOAK_USER': 'admin'}
 KEYCLOAK_USER
 {'KEYCLOAK_USER': 'admin'}

EDIT:

I solved it: I had to call the items() function.

{% for envvar in list_of_envvars %}
{% set envvars_dict = lookup('vars', envvar) %}
{% for key, value in envvars_dict.items() %}
{% if value %}
{{key}}={{value}}
{% endif %}
{% endfor %}
{% endfor %}