Managing "nested" group in Ansible YAML inventory files

Building up on input in latest @larsk's answer with the constructed inventory plugin. Original answer below

There is a possibility to DRY a bit more with the constructed plugin if you keep the same naming convention for cluster groups/subgroups, eliminating the need to introduce an additional variable in the groups.

Note that using convention for file names inside the inventory dir allows for natural loading order of the inventory sources as well without having to modify ansible.cfg

Given the following minimal inventories/cluster/cluster.yml static inventory source:

---
all:
  children:
    cluster_one:
      children:
        cluster_one_controller:
          hosts:
            host1:
            host2:
            host3:
        cluster_one_compute:
          hosts:
            host4:
            host5:
            host6:
    cluster_two:
      children:
        cluster_two_controller:
          hosts:
            host11:
            host12:
            host13:

And the corresponding inventories/cluster/cluster_constructed.yml dynamic inventory source based on existing group names detection:

---
plugin: constructed
strict: false
groups:
  controller: group_names | select('match', '^.*_controller$') | length > 0
  compute: group_names | select('match', '^.*_compute$') | length > 0

We get the expected result using our composite inventory directory

$ ansible-inventory -i inventories/cluster --list
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "cluster_one",
            "cluster_two",
            "compute",
            "controller",
            "ungrouped"
        ]
    },
    "cluster_one": {
        "children": [
            "cluster_one_compute",
            "cluster_one_controller"
        ]
    },
    "cluster_one_compute": {
        "hosts": [
            "host4",
            "host5",
            "host6"
        ]
    },
    "cluster_one_controller": {
        "hosts": [
            "host1",
            "host2",
            "host3"
        ]
    },
    "cluster_two": {
        "children": [
            "cluster_two_controller"
        ]
    },
    "cluster_two_controller": {
        "hosts": [
            "host11",
            "host12",
            "host13"
        ]
    },
    "compute": {
        "hosts": [
            "host4",
            "host5",
            "host6"
        ]
    },
    "controller": {
        "hosts": [
            "host1",
            "host11",
            "host12",
            "host13",
            "host2",
            "host3"
        ]
    }
}

original answer

I'm deeply sorry this made you sad @larks. Unfortunately, as you just experienced:

• the content of the group does not depend on the place where you define it and will at the end be a consolidation of all hosts/children definitions.
• the group defined as a children anywhere will contain all the hosts defined elsewhere.

The only static yaml inventory definition I can think of that gets as close to your initial requirement and respects the DRY principle as much as possible is:

---
all:
  children:
    cluster_one:
      children:
        cluster_one_controller:
          hosts:
            host1:
            host2:
            host3:
        cluster_one_compute:
          hosts:
            host4:
            host5:
            host6:
    cluster_two:
      children:
        cluster_two_controller:
          hosts:
            host11:
            host12:
            host13:
    controller:
      children:
        cluster_one_controller:
        cluster_two_controller:
    compute:
      children:
        controller_one_compute:

This will let you select any of the following patterns (non exhaustive list):

• a full cluster e.g. cluster_one
• all controllers of a cluster e.g cluster_two_controller or cluster_two:&controller
• all compute nodes across clusters e.g. compute
• ...

Hope this will help to shine some light on your day (or night...)

By using an inventory layout similar to what @Zeitounator suggested and making use of the constructed inventory plugin, I've been able to get what I want. I start with a static inventory in inventory/static/clusters.yml that looks like:

all:
  children:
    cluster_one:
      children:
        cluster_one_controller:
          vars:
            node_role: controller
          hosts:
            host1:
            host2:
            host3:
        cluster_one_compute:
          vars:
            node_role: compute
          hosts:
            host4:
            host5:
            host6:
    cluster_two:
      children:
        cluster_two_controller:
          vars:
            node_role: controller
          hosts:
            host11:
            host12:
            host13:

Then I mix in the following inventory from inventory/dynamic/constructed.yml:

plugin: constructed
strict: false
keyed_groups:
  - prefix: ""
    separator: ""
    key: node_role

And an ansible.cfg that looks like:

[defaults]
inventory = inventory/static,inventory/dynamic

(That ensures the constructed inventory loads after the static inventory, without mucking about trying to manually order filenames).

With the above in place, I can run a task on just cluster_one:

ansible cluster_one ...

Or on all controllers:

ansible controller ...

Or just on controllers in cluster_two:

ansible 'cluster_two:&controller' ...

Etc.

(For that last example I can of course write cluster_two_controller instead of cluster_two:&controller, but I like the intersection syntax because it matches my mental map of what I'm trying to do.)

How much does it take to code it in a play? Given the structure in the dictionary

shell> cat _hosts
all:
  cluster_one:
    controller:
      - host1
      - host2
      - host3
    compute:
      - host4
      - host5
      - host6
  cluster_two:
    controller:
      - host11
      - host12
      - host13

The play creates the dictionary of the groups

- hosts: localhost
  vars:
    inv: "{{ lookup('file', '_hosts')|from_yaml }}"
  tasks:
    - name: all
      set_fact:
        _groups: "{{ _groups|d({})|combine({'all': _hosts}) }}"
      vars:
        _hosts: "{{ inv.all|json_query('*.*')|flatten }}"
    - name: clusters
      set_fact:
        _groups: "{{ _groups|combine({item: _hosts}) }}"
      loop: "{{ inv.all.keys()|list }}"
      vars:
        _hosts: "{{ inv.all|json_query(item ~ '.*')|flatten }}"
    - name: sub_clusters
      set_fact:
        _groups: "{{ _groups|combine(item, list_merge='append') }}"
      loop: "{{ inv.all|json_query('*') }}"
    - debug:
        var: _groups

gives

  _groups:
    all:
    - host1
    - host2
    - host3
    - host4
    - host5
    - host6
    - host11
    - host12
    - host13
    cluster_one:
    - host1
    - host2
    - host3
    - host4
    - host5
    - host6
    cluster_two:
    - host11
    - host12
    - host13
    compute:
    - host4
    - host5
    - host6
    controller:
    - host1
    - host2
    - host3
    - host11
    - host12
    - host13

Then create the groups

    - add_host:
        name: "{{ item.1 }}"
        groups: "{{ item.0.key }}"
      with_subelements:
        - "{{ _groups|dict2items }}"
        - value
    - debug:
        var: groups

gives

  groups:
    all:
    - host1
    - host2
    - host3
    - host4
    - host5
    - host6
    - host11
    - host12
    - host13
    cluster_one:
    - host1
    - host2
    - host3
    - host4
    - host5
    - host6
    cluster_two:
    - host11
    - host12
    - host13
    compute:
    - host4
    - host5
    - host6
    controller:
    - host1
    - host2
    - host3
    - host11
    - host12
    - host13
    ungrouped: []