Python chunking with regular expressions

user_169947 0
Viewed 159

In Perl, it was easy to iterate over a string to chunk it into tokens:

$key = ".foo[4][5].bar.baz";

@chunks = $key =~ m/\G\[\d+\]|\.[^][.]+/gc;
print "@chunks\n";
#>> output: .foo [4] [5] .bar .baz

# Optional error handling:
die "Malformed key at '" . substr($key, pos($key)) . "'"
  if pos($key) != length($key);

If more control is needed, that can be turned into a loop instead:

while ($key =~ m/(\G\[\d+\]|\.[^][.]+)/g) {
  push @chunks, $1;  # Optionally process each one
}

I'd like to find a clean, idiomatic way to do this in Python. So far I only have this:

import re

key = ".foo[4][5].bar.baz"

rx = re.compile(r'\[\d+\]|\.[^][.]+')
chunks = []
while True:
    m = re.match(rx, key)
    if not m:
        raise ValueError(f"Malformed key at '{key}'")
    chunk = m.group(0)
    chunks.append(chunk[1:] if chunk.startswith('.') else int(chunk[1:-1]))
    key = key[m.end(0):]

    if key == '':
        break

print(chunks)

Aside from it being a lot more verbose, I don't love that because I need to destroy the string as I process it, since there doesn't seem to be an equivalent of Perl's \G anchor (pick up where the last match left off). An alternative would be to keep track of my own match position in the string in each loop, but that seems even more fiddly.

Is there some idiom I haven't found? I also tried some solution using re.finditer() but it doesn't seem to have a way to have each match start at the exact end of the previous match (e.g. re.matchiter() or somesuch).

Suggestions & discussion welcome.

tokenize python regex
1 Answers

Summary

There is no direct equivalent of re.matchiter() as you described it.

Two alternatives come to mind:

  1. Create a mismatch token.
  2. Write your own generator with the desired behavior.

Mismatch Token

The usual technique in Python is to define a MISMATCH catchall token and to raise an exception if that token is ever encountered.

Here's a working example (one that I wrote and put in the Python docs so that everyone could find it):

from typing import NamedTuple
import re

class Token(NamedTuple):
    type: str
    value: str
    line: int
    column: int

def tokenize(code):
    keywords = {'IF', 'THEN', 'ENDIF', 'FOR', 'NEXT', 'GOSUB', 'RETURN'}
    token_specification = [
        ('NUMBER',   r'\d+(\.\d*)?'),  # Integer or decimal number
        ('ASSIGN',   r':='),           # Assignment operator
        ('END',      r';'),            # Statement terminator
        ('ID',       r'[A-Za-z]+'),    # Identifiers
        ('OP',       r'[+\-*/]'),      # Arithmetic operators
        ('NEWLINE',  r'\n'),           # Line endings
        ('SKIP',     r'[ \t]+'),       # Skip over spaces and tabs
        ('MISMATCH', r'.'),            # Any other character
    ]
    tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification)
    line_num = 1
    line_start = 0
    for mo in re.finditer(tok_regex, code):
        kind = mo.lastgroup
        value = mo.group()
        column = mo.start() - line_start
        if kind == 'NUMBER':
            value = float(value) if '.' in value else int(value)
        elif kind == 'ID' and value in keywords:
            kind = value
        elif kind == 'NEWLINE':
            line_start = mo.end()
            line_num += 1
            continue
        elif kind == 'SKIP':
            continue
        elif kind == 'MISMATCH':
            raise RuntimeError(f'{value!r} unexpected on line {line_num}')
        yield Token(kind, value, line_num, column)

statements = '''
    IF quantity THEN
        total := total + price * quantity;
        tax := price * 0.05;
    ENDIF;
'''

for token in tokenize(statements):
    print(token)

Custom Generator

Another alternative is to write a custom generator with the desired behavior.

The match() method for compiled regular expressions allows an optional starting position for the match operation. With that tool, it isn't hard to write a custom generator that applies match() to consecutive starting positions:

def itermatch(pattern, string):
    p = re.compile(pattern)
    pos = 0
    while True:
        mo = p.match(string, pos)
        if mo is None:
            break             # Or raise exception
        yield mo
        pos = mo.end()
user_4244990
Related