ANTRL4 grammar cannot cover all cases

user_10266230 0
Viewed 61
// Define a grammar called Hello
grammar Hello;

r  : element* ;         

element
    : number Whitespace
    | string Whitespace
    ;

string
    : '(' Charactor* ')'
    ;

Charactor 
    : [a-zA-Z] |'!' | '"'| '#' | '$' | '%' | '&' | '\'' | '\\(' | '\\)' | '*' | '+' | ',' | '-' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | '\\' | ']' | '^' | '`' | '{' | '|' | '}' | '~' | '_'
    ;

number
    : '-'? integer ('.' integer)?
    ;

integer
    : digit+
    ;

digit
    : D0 | D1 | D2 | D3 | D4 | D5 | D6 | D7 | D8 | D9
    ;


Whitespace
    : ' ' | '\n'
    ;

D1 : '1';
D2 : '2';
D3 : '3';
D4 : '4';
D5 : '5';
D6 : '6';
D7 : '7';
D8 : '8';
D9 : '9';
D0 : '0';

//WS : [ \t\r\n]+ -> skip ; skip spaces, tabs, newlines

I used the above .g4 grammar file to parse the following sequence of characters.

➜ Hello antlr4 Hello.g4 ➜ Hello javac Hello*.java ➜ Hello grun Hello r -tree 1.1 -1.2 333 -222 (((*&^%$#@!~<>,?"'\|[[]]{}~) (r (element (number (integer (digit 1)) . (integer (digit 1))) ) (element (number - (integer (digit 1)) . (integer (digit 2))) ) (element (number (integer (digit 3) (digit 3) (digit 3))) ) (element (number - (integer (digit 2) (digit 2) (digit 2))) ) (element (string ( \( \( * & ^ % $ # @ ! ~ < > , ? " ' \ | [ [ ] ] { } ~ )) ))

This case works well.

But when I input the string "1.1 -1.2 333 -222 (-.#$?)", it doesn't parse it correctly.

➜ Hello grun Hello r -tree 1.1 -1.2 333 -222 (-.#$?) line 1:19 mismatched input '-' expecting {')', Charactor} line 1:20 mismatched input '.' expecting {'1', '2', '3', '4', '5', '6', '7', '8', '9', '0'} line 1:21 mismatched input '#' expecting {'1', '2', '3', '4', '5', '6', '7', '8', '9', '0'} (r (element (number (integer (digit 1)) . (integer (digit 1))) ) (element (number - (integer (digit 1)) . (integer (digit 2))) ) (element (number (integer (digit 3) (digit 3) (digit 3))) ) (element (number - (integer (digit 2) (digit 2) (digit 2))) ) (element (string () ) (element (number - integer . (integer # $ ? ))) )) ➜ Hello

This special sequence of characters is an array that includes numbers or strings. The number can come with these ways like 1.1, -1.2, -222, 222 The string starts with '(', and ends with ')', if '(' or ')' appear in it, it could be escaped. It's noted that the string can include the character '-' or '.'. So when '-' or '.' appears in the numbers and strings at the same time, it seems Antlr cannot parse it correctly.

Does anyone know how to fix this issue? Thanks!

antlr4 context-free-grammar java
1 Answers

ANTLR has a strict separation between parser- and lexer rules. Whenever you use literal tokens in parser rules (like '-' and '.' in the number rule), ANTLR creates lexer rules behind the scenes for you. So the grammar:

number
    : '-'? integer ('.' integer)?
    ;

Charactor 
    : [a-zA-Z] | ... | '-' | '.' | ...
    ;

really looks like this:

number
    : T_0? integer (T_1 integer)?
    ;

T_0 : '-';

T_1 : '.';

Charactor 
    : [a-zA-Z] | ... | '-' | '.' | ...
    ;

And since the strict separation, ANTLR creates tokens independently from the parser. This means that for the characters - and ., it will never happen that these become Charactor tokens. They will always become T_0 and T_1 tokens. There is no way around this. If you freely want to used any character/token in any rule, look for "scannerless parsing" or "PEG parser" instead of using ANTLR.

To make your current grammar work with as little changes as possible, do something like this:

grammar Hello;

r  : element* ;

element
    : number Whitespace
    | string Whitespace
    ;

string
    : '(' (Charactor | Minus | Dot)* ')'
    ;

Minus : '-';
Dot : '.';

Charactor
    : [a-zA-Z] |'!' | '"'| '#' | '$' | '%' | '&' | '\'' | '\\(' | '\\)' | '*' | '+' | ',' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | '\\' | ']' | '^' | '`' | '{' | '|' | '}' | '~' | '_'
    ;

number
    : Minus? integer (Dot integer)?
    ;

integer
    : digit+
    ;

digit
    : D0 | D1 | D2 | D3 | D4 | D5 | D6 | D7 | D8 | D9
    ;


Whitespace
    : ' ' | '\n'
    ;

D1 : '1';
D2 : '2';
D3 : '3';
D4 : '4';
D5 : '5';
D6 : '6';
D7 : '7';
D8 : '8';
D9 : '9';
D0 : '0';

But ideally, you'd use ANTLR more as it is intended to be (clearly defining tokens instead of constructing tokens in parser rules):

grammar Hello;

r  : element* EOF;

element
    : Number Whitespace
    | string Whitespace
    ;

string
    : '(' character* ')'
    ;

character
    : Character
    | Minus
    | Dot
    ;

Minus
    : '-'
    ;

Dot
    : '.'
    ;

Character
    : [a-zA-Z] | '!' | '"'| '#' | '$' | '%' | '&' | '\'' | '\\(' | '\\)' | '*' | '+' | ',' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | '\\' | ']' | '^' | '`' | '{' | '|' | '}' | '~' | '_'
    ;

Number
    : '-'? Integer ('.' Integer)?
    ;

Integer
    : Digit+
    ;

Whitespace
    : [ \t\r\n]
    ;

fragment Digit
    : [0-9]
    ;

A few tips:

  • do not mix lexer and parser rules: start with parser rules, followed by lexer rules
  • do not use literal tokens (like '.') inside parser rules

If you do that, it will be more clear when a token will be chosen over another token: the lexer:

  1. matches rules from top to bottom, always choosing the longest possible match, and
  2. when 2 (or more) lexer rules match the same amount of characters, the rule defined first "wins"

That is why (in my second grammar) the input "-1" will become a single Number token, and not a Minus token followed by a Number token (longest match "wins").

user_504760
Related