How to gather config files in a list with Hydra-fb?

user_10484775 0
Viewed 1680

Say I have an abstract class db in my code and classes db1, db1, ... db1 that inherit from db. My project uses hydra and has this structure:

├── my_app.py
├── conf.yaml
└── db
    ├── db1.yaml
    ├── db2.yaml
    └── db3.yaml

I need a list of db so I would like to get a final configuration file like this :

db:
  -
    param1_of_db1: key_1_1
    param2_of_db1: key_1_2
  -
    param1_of_db2: key_2_1
    param2_of_db2: key_2_2
  -
    param1_of_db3: key_3_1
    param2_of_db3: key_3_2

so that db is a list of params of db1, db2, db3. In conf.yaml file, I imagine something like:

defaults:
  - db: [db1, db2, db3]

Is there a way to do something like that ?

fb-hydra python
2 Answers

What you are asking for is not supported by Hydra.

  1. List composition is not supported, lists are all or nothing during composition.
  2. Config groups are mutually exclusive, there is a feature request to relax that.

You can get close to it though (without being able to override the structure from the command line is something like: config.yaml:

defaults:
  - db/db1
  - db/db2
  - db/db3

This syntax is documented here.

In each db config file you can do something like:

db/db1.yaml:

# @pacakge _group_._name_
host: localhost
port: 3306

Package overrides are documented here.

The resulting config would looke like:

db:       # from the config group of the corresponding config (path)
  db1:    # from the name of the corresponding config
    host: localhost
    port: 3306
  db2:
    ...
user_19308380

You can get pretty close to list merging with the latest hydra/omegaconf

The trick is that dictionaries are composed, so you can compose the config in a dictionary and then use the new oc.dict.values interpolation to get the final list.

So in your case it would be something like:

defaults:
  - dbs/db1
  - dbs/db2
  - dbs/db3

db: ${oc.dict.values:dbs}

Note that I've renamed the "db" package to "dbs". So the final resolved config would look like:

dbs:
  db1:    
    host: localhost
    port: 3306
  db2:
    host: localhost
    port: 3307
  db3:
    host: localhost
    port: 3308

db:
  - host: localhost
    port: 3306
  - host: localhost
    port: 3307
  - host: localhost
    port: 3308

The keys for the "dbs" package aren't used for anything but I find that it actually makes the config clearer to have them.

user_5645400
Related