how to reference a YAML "setting" from elsewhere in the same YAML file?

Viewed 176118

I have the following YAML:

paths:
  patha: /path/to/root/a
  pathb: /path/to/root/b
  pathc: /path/to/root/c

How can I "normalise" this, by removing /path/to/root/ from the three paths, and have it as its own setting, something like:

paths:
  root: /path/to/root/
  patha: *root* + a
  pathb: *root* + b
  pathc: *root* + c

Obviously that's invalid, I just made it up. What's the real syntax? Can it be done?

10 Answers

I don't think it is possible. You can reuse "node" but not part of it.

bill-to: &id001
    given  : Chris
    family : Dumars
ship-to: *id001

This is perfectly valid YAML and fields given and family are reused in ship-to block. You can reuse a scalar node the same way but there's no way you can change what's inside and add that last part of a path to it from inside YAML.

If repetition bother you that much I suggest to make your application aware of root property and add it to every path that looks relative not absolute.

YML definition:

dir:
  default: /home/data/in/
  proj1: ${dir.default}p1
  proj2: ${dir.default}p2
  proj3: ${dir.default}p3 

Somewhere in thymeleaf

<p th:utext='${@environment.getProperty("dir.default")}' />
<p th:utext='${@environment.getProperty("dir.proj1")}' /> 

Output: /home/data/in/ /home/data/in/p1

In some languages, you can use an alternative library, For example, tampax is an implementation of YAML handling variables:

const tampax = require('tampax');

const yamlString = `
dude:
  name: Arthur
weapon:
  favorite: Excalibur
  useless: knife
sentence: "{{dude.name}} use {{weapon.favorite}}. The goal is {{goal}}."`;

const r = tampax.yamlParseString(yamlString, { goal: 'to kill Mordred' });
console.log(r.sentence);

// output : "Arthur use Excalibur. The goal is to kill Mordred."

Editor's Note: poster is also the author of this package.

With Yglu, you can write your example as:

paths:
  root: /path/to/root/
  patha: !? .paths.root + a
  pathb: !? .paths.root + b
  pathc: !? .paths.root + c

Disclaimer: I am the author of Yglu.

Using OmegaConf

OmegaConf is a YAML-based hierarchical configuration system that has support for this under the functionality Variable interpolation. Using OmegaConf v2.2.2:

Create a YAML file paths.yaml as follows:

paths:
  root: /path/to/root/
  patha: ${.root}a
  pathb: ${.root}b
  pathc: ${.root}c

then we can read the file with variable paths:

from omegaconf import OmegaConf
conf = OmegaConf.load("test_paths.yaml")

>>> conf.paths.root
'/path/to/root/'

>>> conf.paths.patha
'/path/to/root/a'
>>> conf.paths.pathb
'/path/to/root/b'
>>> conf.paths.pathc
'/path/to/root/c'

Deep and Cross -Ref

It is possible to define more complex (nested) structures with a relative depth of your variable in reference to other variables:

Create another file nested_paths.yaml:

data:
    base: data
    sub_dir_A:
        name: a
        # here we note that `base` is two levels above this variable
        # hence we will use `..base` two dots but the `name` variable is
        # at the same level hence a single dot `.name`
        nested_dir: ${..base}/sub_dir/${.name}/last_dir 
    sub_dir_B:
        # add another level of depth
        - name: b
          # due to another level of depth, we have to use three dots
          # to access `base` variable as `...base`
          nested_file: ${...base}/sub_dir/${.name}/dirs.txt
        - name: c
          # we can also make cross-references to other variables
          cross_ref_dir: ${...sub_dir_A.nested_dir}/${.name}

again we can check:

conf = OmegaConf.load("nested_paths.yaml")

# 1-level of depth reference
>>> conf.data.sub_dir_A.nested_dir
'data/sub_dir/a/last_dir'

# 2-levels of depth reference
>>> conf.data.sub_dir_B[0].nested_file
'data/sub_dir/b/dirs.txt'

# cross-reference example
>>> conf.data.sub_dir_B[1].cross_ref_dir
'data/sub_dir/a/last_dir/c'

In case of invalid references (such as wrong depth, wrong variable name), OmegaConf will throw an error omegaconf.errors.InterpolationResolutionError. It is also used in Hydra for configuring complex applications.

I have written my own library on Python to expand variables being loaded from directories with a hierarchy like:

/root
 |
 +- /proj1
     |
     +- config.yaml
     |
     +- /proj2
         |
         +- config.yaml
         |
         ... and so on ...

The key difference here is that the expansion must be applied only after all the config.yaml files is loaded, where the variables from the next file can override the variables from the previous, so the pseudocode should look like this:

env = YamlEnv()
env.load('/root/proj1/config.yaml')
env.load('/root/proj1/proj2/config.yaml')
...
env.expand()

As an additional option the xonsh script can export the resulting variables into environment variables (see the yaml_update_global_vars function).

The scripts:

https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/python/cmdoplib/cmdoplib.yaml.xsh https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/python/tacklelib/tacklelib.yaml.py

Pros:

  • simple, does not support recursion and nested variables
  • can replace an undefined variable to a placeholder (${MYUNDEFINEDVAR} -> *$/{MYUNDEFINEDVAR})
  • can expand a reference from environment variable (${env:MYVAR})
  • can replace all \\ to / in a path variable (${env:MYVAR:path})

Cons:

  • does not support nested variables, so can not expand values in nested dictionaries (something like ${MYSCOPE.MYVAR} is not implemented)
  • does not detect expansion recursion, including recursion after a placeholder put
Related