YAML anchor for sequence?

Viewed 945

Consider the following file YAML file:

A:
  &B
  - x: 1
    y: 2
  - x: 10
    y: 20

C:
  <<: *B

which is read into python via:

from ruamel.yaml import YAML
filename = 'data/configs/debug_config.yml'

with open(filename) as f:
    c = YAML(typ='safe').load(f)

print(c)

yielding:

{'A': [{'x': 1, 'y': 2}, {'x': 10, 'y': 20}], 'C': {'x': 1, 'y': 2}}

It is seen that the anchor B only includes the first element of the sequence. Why? I would like an anchor that would include the entire sequence, such that the values of A and C in the python dictionary are identical. How can this be done?

2 Answers

The anchor B includes all elements from A, but you are merging them with the merge key << (source):

If the value associated with the merge key is a sequence, then this sequence is expected to contain mapping nodes and each of these nodes is merged in turn according to its order in the sequence. Keys in mapping nodes earlier in the sequence override keys specified in later mapping nodes.

So,the first item from A overrides the second one.

Remove <<: and C will be the same dictionary as A:

A: &B
  - x: 1
    y: 2
  - x: 10
    y: 20

C: *B

Yields:

{
  "A": [
    {
      "y": 2, 
      "x": 1
    }, 
    {
      "y": 20, 
      "x": 10
    }
  ], 
  "C": [
    {
      "y": 2, 
      "x": 1
    }, 
    {
      "y": 20, 
      "x": 10
    }
  ]
}

You can check the merge order with this example:

A: &B
  - x: 1
    y: 2
  - x: 10
    new: 333 


C: 
  <<: *B

Yields:

{
  "A": [
    {
      "y": 2, 
      "x": 1
    }, 
    {
      "x": 10, 
      "new": 333
    }
  ], 
  "C": {
    "y": 2, 
    "x": 1, 
    "new": 333
  }
}

As @Hadyniak already indicated you are incorrectly using the merge keys. Since the alias *B is expanded in the composer step, before the interpretation of the << merge key in the constructor step, the latter actually receives a list of dicts, which are combined with the values of keys of earlier occuring dicts taking precedence. If the order of processing had happened to be different you would get an error, and IMO the merge key documentation doesn't explicitly specify that aliases should be expanded first.

Hadyniak's solution however will have you end up with c['A'] and c['C'] being the same dictionary and that might not be what you want:

import ruamel.yaml

yaml_str = """\
A:
  &B
  - x: 1
    y: 2
  - x: 10
    y: 20

C:
  *B
"""

yaml = ruamel.yaml.YAML()
c = yaml.load(yaml_str)
print(c['A'] is c['C'], c['A'] == c['C'])
c['A'][0]['x'] = 42
print('x:', c['C'][0]['x'])

which gives:

True True
x: 42

If that is not what you want, you can still use merge keys, but on the dictionaries that are the elements of your sequence:

import ruamel.yaml

yaml_str = """\
A:
  - &B1
    x: 1
    y: 2
  - &B2
    x: 10
    y: 20

C:
  - <<: *B1
  - <<: *B2
"""

yaml = ruamel.yaml.YAML()
c = yaml.load(yaml_str)
print(c['A'] is c['C'], c['A'] == c['C'])
c['A'][0]['x'] = 42
print('x:', c['C'][0]['x'])

which gives:

False True
x: 1

Alternatively you can tell the composer part of ruamel.yaml to expand the aliases instead of using a reference:

import copy

yaml_str = """\
A:
  &B
  - x: 1
    y: 2
  - x: 10
    y: 20

C:
  *B
"""

yaml = ruamel.yaml.YAML()

yaml.composer.return_alias = lambda s: copy.deepcopy(s)

c = yaml.load(yaml_str)
print(c['A'] is c['C'], c['A'] == c['C'])
c['A'][0]['x'] = 42
print('x:', c['C'][0]['x'])

which gives:

False True
x: 1

The above will only work in ruamel.yaml>0.17.2, for older versions you'll need to copy and modify the compose_node method:

import copy

yaml_str = """\
A:
  &B
  - x: 1
    y: 2
  - x: 10
    y: 20

C:
  *B
"""

class MyComposer(ruamel.yaml.composer.Composer):
    def compose_node(self, parent, index):
        # type: (Any, Any) -> Any
        if self.parser.check_event(ruamel.yaml.events.AliasEvent):
            event = self.parser.get_event()
            alias = event.anchor
            if alias not in self.anchors:
                raise ComposerError(
                    None,
                    None,
                    'found undefined alias {alias!r}'.format(alias=alias),
                    event.start_mark,
                )
            return copy.deepcopy(self.anchors[alias])
        event = self.parser.peek_event()
        anchor = event.anchor
        if anchor is not None:  # have an anchor
            if anchor in self.anchors:
                ws = (
                    '\nfound duplicate anchor {!r}\nfirst occurrence {}\nsecond occurrence '
                    '{}'.format((anchor), self.anchors[anchor].start_mark, event.start_mark)
                )
                warnings.warn(ws, ruamel.yaml.error.ReusedAnchorWarning)
        self.resolver.descend_resolver(parent, index)
        if self.parser.check_event(ruamel.yaml.events.ScalarEvent):
            node = self.compose_scalar_node(anchor)
        elif self.parser.check_event(ruamel.yaml.events.SequenceStartEvent):
            node = self.compose_sequence_node(anchor)
        elif self.parser.check_event(ruamel.yaml.events.MappingStartEvent):
            node = self.compose_mapping_node(anchor)
        self.resolver.ascend_resolver()
        return node



yaml = ruamel.yaml.YAML()
yaml.Composer = MyComposer

c = yaml.load(yaml_str)
print(c['A'] is c['C'], c['A'] == c['C'])
c['A'][0]['x'] = 42
print('x:', c['C'][0]['x'])

which also gives:

False True
x: 1
Related