Ansible include_vars into dictionary

In my ansible playbook, I read a list of directories into a list. I then want to read a "config.yml" file from each of these directories and put their content into dictionary, so that I can reference the config-data via the directory name from that dictionary. The first part is no problem, but I cannot get the second part to work:

Step 1, load directories:

- name: Include directories
  include_vars:
  file: /main-config.yml
  name: config

Step 2, load configs from directories:

- name: load deploymentset configurations
  include_vars:
    file: /path/{{ item }}/config.yml
    name: "allconfs.{{ item }}"   ## << This is the problematic part 
  with_items:
    - "{{ config.dirs }}"

I tried different things like "allconfs['{{ item }}'], but none seemed to work. The playbook completed successfully, but the data was not in the dictionary. I also tried defining the outer dictionary beforehand, but that did not work either.

The config files themselves are very simple:

/main-config.yml:

dirs:
- dir1
- dir2
- dir3

/path/dir1/config.yml:

some_var: "some_val"
another_var: "another val"

I want to be able to then access the values of the config.yml files like this:

{{ allconfs.dir1.some_var }}

UPDATE to try Konstantins approach:

  - name: load deploymentset configurations
    include_vars:
      file: /repo/deploymentsets/{{ item }}/config.yml
      name: "default_config"
    with_items:
    - "{{ config.deploymentsets }}"
    register: default_configs


  - name: combine configs
    set_fact:
      default_configs: "{{ dict(default_configs.results | json_query('[].[item, ansible_facts.default_config]')) }}"

Error message:

fatal: [127.0.0.1]: FAILED! => {"failed": true, "msg": "Unexpected templating type error occurred on ({{ dict(default_configs.results | json_query('[].[item, ansible_facts.default_config]')) }}): <lambda>() takes exactly 0 arguments (1 given)"}