How to parse literal block with user supplied START and END token

Viewed 45

I am trying to parse a language that has a block with user supplied start/end mark, see following example

@IDENT1
text... text...
text ...
IDENT1

OR

@@IDENT2
text... text...
text ...
IDENT2

are both valid block, but following is not

@IDENT1
text ... text...
IDENT2

How could I write a rule in ANTLR4 to parse this?

1 Answers

@BartKiers example in any language is fine

OK, here's a demo for the Java runtime:

lexer grammar BlockLexer;

@members {
  private boolean blockEndAhead(String matchedText) {

    if (this._input.LA(1) == EOF) {
      // There apaprently is no proper closing block ahead, because we've stumbled upon the end-of-file token
      return false;
    }

    String startBlock = matchedText.replaceAll("^@+(\\w+)[\\s\\S]+$", "$1");

    for (int ahead = 1; ahead <= startBlock.length(); ahead++) {
      char charAhead = (char)this._input.LA(ahead);

      if (charAhead != startBlock.charAt(ahead - 1)) {
        return false;
      }
    }

    return true;
  }
}

BLOCK
 : '@'+ IDENTIFIER [ \t\r\n]+
   // Match as much chars as possible until we "see" the opening ID ahead in the char stream
   ({!this.blockEndAhead(getText())}? . )*
   // Make sure the proper closing ID is ahead, if not, the lexer will fall through to `INVALID_BLOCK`
   {this.blockEndAhead(getText())}? IDENTIFIER
 ;

// Match a dangling open-ID that has no closing ID
INVALID_BLOCK
 : '@'+ IDENTIFIER [ \t\r\n]+ ({!this.blockEndAhead(getText())}? . )*
 ;

// Fall through rule: matches any single char if none of the lexer rules above have matched
UNKNOWN : . ;

fragment IDENTIFIER
 : [a-zA-Z_] [a-zA-Z_0-9]*
 ;

Given the test input:

@IDENT1
text... text...
text ...
IDENT1

noise

@@IDENT2
text... text...
text ...
IDENT2

noise

@IDENT1
text ... text...
IDENT2

and if you generate a lexer and run the following Java code:

String source = "@IDENT1\n" +
    "text... text...\n" +
    "text ...\n" +
    "IDENT1\n" +
    "\n" +
    "noise\n" +
    "\n" +
    "@@IDENT2\n" +
    "text... text...\n" +
    "text ...\n" +
    "IDENT2\n" +
    "\n" +
    "noise\n" +
    "\n" +
    "@IDENT1\n" +
    "text ... text...\n" +
    "IDENT2";

BlockLexer lexer = new BlockLexer(CharStreams.fromString(source));
CommonTokenStream stream = new CommonTokenStream(lexer);
stream.fill();

for (Token t : stream.getTokens()) {
  System.out.printf("%-35s `%s`%n",
      BlockLexer.VOCABULARY.getSymbolicName(t.getType()),
      t.getText().replace("\n", "\\n"));
}

the following will be printed:

BLOCK                               `@IDENT1\ntext... text...\ntext ...\nIDENT1`
UNKNOWN                             `\n`
UNKNOWN                             `\n`
UNKNOWN                             `n`
UNKNOWN                             `o`
UNKNOWN                             `i`
UNKNOWN                             `s`
UNKNOWN                             `e`
UNKNOWN                             `\n`
UNKNOWN                             `\n`
BLOCK                               `@@IDENT2\ntext... text...\ntext ...\nIDENT2`
UNKNOWN                             `\n`
UNKNOWN                             `\n`
UNKNOWN                             `n`
UNKNOWN                             `o`
UNKNOWN                             `i`
UNKNOWN                             `s`
UNKNOWN                             `e`
UNKNOWN                             `\n`
UNKNOWN                             `\n`
INVALID_BLOCK                       `@IDENT1\ntext ... text...\nIDENT2`
EOF                                 `<EOF>`
Related