How can I distinguish end of line comments from full line comments in ruamel.yaml?

user_2349661 0
Viewed 108

In an attempt to make complex Azure DevOps pipelines self-documenting I am trying to read comments out of YAML files automatically. I have decided to use the ruamel.yaml python library.

Reading comments works well, but I have not found how to distinguish comments at the end of the line from full-line comments:

  - book # This is an end-of-the-line comment
    # This is a full line comment

Does anyone know how I could achieve this?

Code example, reading the stage-level comments of all stages, without comments of sub-entities of stages:

from ruamel.yaml import YAML
from ruamel.yaml.comments import CommentedMap, CommentedSeq

file_name = 'test.yaml'

yaml=YAML()
with open(file_name) as doc:
    data = yaml.load(doc)

i = 0
for item in data['stages']:
    i+=1
    print("*** stage", i, item['stage'])
    if isinstance(item, CommentedMap):
        comment_token = item.ca.items.get('stage')
        stage_help = {"stage_id": i}
        current_key = "Comment"
        if comment_token:
            for tab in comment_token:
                if tab:
                    vals = tab.value.split('\n')

                    for line in vals:
                        if line[1:1] == "#":
                            line = line[1:]
                        else:
                            line = line.strip()[1:].strip()
                        if len(line) == 0:
                            continue
                        if line[0:1] == "@":
                            current_key = line[1:line.index(':')]
                            content = line[line.index(':')+1:].strip()
                        else:
                            content = line
                        if current_key not in stage_help:
                            stage_help[current_key] = f"{content}"
                        else:
                            stage_help[current_key] = f"{stage_help[current_key]}\n{content}"

                    print(stage_help)

YAML:

stages:
  - stage: TestA
    # @Comment: I write what it does
    # @Link: https://documentation
  - stage: TestB # My favorite stage!
    # @Comment: We can also write
    # Multiline docs
    # @Link: https://doc2
    displayName: Test B # The displayName is shown in the pipeline's GUI

Running this gives me:

*** stage 1 TestA
{'stage_id': 1, 'Comment': 'I write what it does', 'Link': 'https://documentation'}
*** stage 2 TestB
{'stage_id': 2, 'Comment': 'My favorite stage!\nWe can also write\nMultiline docs', 'Link': 'https://doc2'}
ruamel.yaml azure-devops python-3.x yaml
1 Answers

First of all, the ruamel.yaml package indicates that:

The 0.17 series will also see changes in how comments are attached during roundtrip. This will result in backwards incompatibilities on the .ca data and it might even be necessary for documented methods that handle comments.

so you should certainly test for the version of the installed ruamel.yaml in your code.

In ruamel.yaml<0.17. Any comment only lines (and empty lines) are attached to the preceding end-of-line comment that is associated with a napping key (or sequence element index). If such a preceding end-of-line comment doesn't exists, it is constructed as just a newline. That is what you should check for

import sys
import ruamel.yaml

assert ruamel.yaml.version_info < (0, 17)

yaml_str = """\
stages:
  - stage: TestA
    # @Comment: I write what it does
    # @Link: https://documentation
  - stage: TestB # My favorite stage!
    # @Comment: We can also write
    # Multiline docs
    # @Link: https://doc2
    displayName: Test B # The displayName is shown in the pipeline's GUI
"""

def comment_splitter(comment_list):
     """ expects the list that is a comment attached to key/index as argument
     returns a tuple containing;
     - the eol comment for the key/index (empty string if not available)
     - a list of, non-empty, full comments lines following the key/index
     """
     token = comment_list[2]
     # print(token)
     eol, rest = token.value.split('\n', 1)
     return eol.strip(), [y for y in [x.strip() for x in rest.split('\n')] if y]

yaml = ruamel.yaml.YAML()
data = yaml.load(yaml_str)
print(comment_splitter(data['stages'][0].ca.items.get('stage')))
print(comment_splitter(data['stages'][1].ca.items.get('stage')))

which gives:

('', ['# @Comment: I write what it does', '# @Link: https://documentation'])
('# My favorite stage!', ['# @Comment: We can also write', '# Multiline docs', '# @Link: https://doc2'])

Because you immediately split the value of the CommentToken (if available) in your program and just skip if the line is of zero length, you probably missed this. If you uncomment the # print(token) this should become clear.

I recommend against abusing YAML comments this way, and instead ask to consider generating the file that azure pipelines expects by extracting the information from a more complete YAML file with a small python program. That is how I overcome the deficiencies of the docker-compose.yaml file format.

Whether you check in the resulting (simpler) YAML, or always generate it on the fly depends on the capabilities of the pipeline cq. how you invoke it.

I would probably start with something like following, and process that:

stages:
  - @Stage
    name: TestA
    comment: I write what it does
    link: https://documentation
  - @Stage
    name: TestB # My favorite stage!
    Comment: |
        We can also write
        Multiline docs
    link: https://doc2
    displayName: Test B # The displayName is shown in the pipeline's GUI

That way you don't have to deal with any changes in the way comments will be attached in future versions of ruamel.yaml.

user_13079050
Related