How can I use the ruamel.yaml rtsc mode?

user_1134951 0
Viewed 52

I've been working on creating a YAML re-formatter based on ruamel.yaml (which you can see here). I'm currently using version 0.17.20.

Cleaning up comments and whitespace has been difficult. I want to:

  • ensure there is only one space before the # for EOL comments
  • align full line comments with the key or item immediately following
  • remove duplicate blank lines so there is at most one blank line

To get closer to achieving that, I have a custom Emitter class where I extend write_comment to adjust the comments just before writing with super().write_comment(...). However, the Emitter does not know about which key or item comes next because comments are generally attached as post comments.

As I've studied the ruamel.yaml code to figure out how to do this, I found the rtsc mode (Round Trip Split Comments) which looks fantastic because it separates EOLComment, BlankLineComment and FullLineComment instead of lumping them together.

From what I can tell, the Parser and Scanner have been adjusted to capture the comments. So, loading is (mostly?) implemented with this "NEWCMNT" implementation. But Emitter.write_comment expects CommentToken instead of comment line numbers, so dumping does not work yet.

If I update my Emitter.write_comment method, is that enough to finish dumping? Or what else might be necessary? In one of my tries, I ran into a sys.exit in ScannedComments.assign_eol() - what else is needed to finish that?

PS: I wouldn't normally ask how to collaborate on StackOverflow, but this is not a bug report or a feature request, and I'm trying/failing to use a new (undocumented) feature, so I'm filing this here instead of sourceforge.

ruamel.yaml yaml
1 Answers

rtsc is work in progress cq work started but unfinished. It's internals will almost certainly change.

Two of the three points you indicate can relatively easy be implemented:

  • set the column of each comment to 0 ( by recursively going over a loaded data structure similar to here ) if the column is before the position of the end of the value on a line, you'll get one space between the value and the column

  • at the same time doing the recursion in the previous point. Take each comment value and do something like:

       value = '\n'.join(line.strip() for line in value.splitlines()
   while '\n\n\n' in value:
        value = value.replace('\n\n\n', '\n\n')

The indentation to the following element is difficult, depends on the data structure etc. Given that these are full line comments, I suggest you do some postprocessing of the YAML document you generate:

  1. find a full line comment, gather full line comments until next line is not full line comment (i.e. some "real" YAML). Since full line comments are in column[0] if the previous stuff is applied, you don't have to track if you are in a (multi-line) literal or folded scalar string where one of the lines happens to start with #

  2. determine number of spaces before real YAML and apply these to the full line comments.

import sys
import ruamel.yaml

yaml_str = """\
   # the following is a example YAML doc
a:
- b: 42


       # collapse multiple empty lines 
  c: |
    # this is not a comment
    it is the first line of a block style literal scalar
    processing this gobbles a newline which doesn't go into a comment
 # that is unless you have a (dedented) comment directly following


  d: 42       # and some non-full line comment
 

  e:   # another one
 # and some more comments to align
     f: glitter in the dark near the Tannhäuser gate
"""
 

def redo_comments(d):
    def do_one(comment):
        if not comment:
            return
        comment.column = 0
        value = '\n'.join(line.strip() for line in comment.value.splitlines()) + '\n'
        while '\n\n\n' in value:
            value = value.replace('\n\n\n', '\n\n')
        comment.value = value

    def do_values(v):
        for x in v:
            for comment in x:
                do_one(comment)

    def do_loc(v):
        if v is None:
            return
        do_one(v[0])
        if not v[1]:
            return
        for comment in v[1]:
            do_one(comment)

    if isinstance(d, dict):
        do_loc(d.ca.comment)
        do_values(d.ca.items.values())
        for val in d.values():
            redo_comments(val)
    elif isinstance(d, list):
        do_values(d.ca.items.values())
        for elem in d:
            redo_comments(elem)

def realign_full_line_comments(s):
    res = []
    buf = []
    for line in s.splitlines(True):
        if not buf:
            if line and line[0] == '#':
                buf.append(line)
            else:
                res.append(line)
        else:
            if line[0] in '#\n':
                buf.append(line)
            else:
                # YAML line, determine indent
                count = 0
                while line[count] == ' ':
                    count += 1
                    if count > len(line):
                        break  # superfluous?
                indent = ' ' * count
                for cline in buf:
                    if cline[0] == '\n':  # empty
                        res.append(cline)
                    else:
                        res.append(indent + cline)
                buf = []
                res.append(line)
    return ''.join(res)


 
yaml = ruamel.yaml.YAML()
# yaml.indent(mapping=4, sequence=4, offset=2)
# yaml.preserve_quotes = True
data = yaml.load(yaml_str)
redo_comments(data)
yaml.dump(data, sys.stdout, transform=realign_full_line_comments)

which gives:

# the following is a example YAML doc
a:
- b: 42

  # collapse multiple empty lines
  c: |
    # this is not a comment
    it is the first line of a block style literal scalar
    processing this gobbles a newline which doesn't go into a comment
  # that is unless you have a (dedented) comment directly following

  d: 42 # and some non-full line comment

  e: # another one
    # and some more comments to align
    f: glitter in the dark near the Tannhäuser gate
user_13079050
Related