writing an antlr grammar where whitespace is sometimes significant

user_14760023 0
Viewed 119

This is a dummy example, my actual language is more complicated:

grammar wordasnumber;

WS: [ \t\n] -> skip;
AS: [Aa] [Ss];
ID: [A-Za-z]+;
NUMBER: [0-9]+;

wordAsNumber: (ID AS NUMBER)* EOF;

In this language, these two strings are legal:

  • seven as 7 eight as 8
  • seven as 7eight as8

Which is exactly what I told it to do, but not what I want. Because ID and AS are both strings of letters, white space is required between them, I would like that second phrase to be a syntax error. I could add some other rule to try and match theses mashed up things ...

fragment LETTER: [A-Za-z];
fragment DIGIT: [0-9];
BAD_THING: ( LETTER+ DIGIT (LETTER|DIGIT)* ) | ( DIGIT+ LETTER (LETTER|DIGIT)* );
ID: LETTER+;
NUMBER: DIGIT+;

... to make the lexer return a different token for these smashed up things, but this feels like a weird bandaid which sort of found the need for accidentally and maybe there are more if I really stared at my lexer very carefully.

Is there a better way to do this? My actual grammar is much larger so, for example, making WS NOT be skipped and placing it explicitly between the tokens where it is required is non starter.

There was an older question on this list, which I could not find, which I think is the same question, in that case someone who was parsing white space separated numbers was surprised that 1.2.3 was parsing as 1.2 and .3 and not as a syntax error.

antlr whitespace
1 Answers

Add another rule for the wrong input, but don't use that in your parser. It will then cause a syntax error when matched:

INVALID: (ID | NUMBER)+;

This additional rule will change the parse tree output, for the input in the question, to:

enter image description here

This trick works because ANTLR4's lexing approach tries to match the longest input in on go, and that INVALID rule matches more than ID and NUMBER alone. But you have to place it after these 2 rules, to make use of another lexing rule: "If two lexer rules would match the same input, pick the first one.". This way, you get the correct tokens for single appearances of ID and NUMBER.

user_11371740
Related