Simple ANTLR grammar not disambiguating identifiers and literals

Question

I'm trying to match this syntax:

P = 100
require P
credit account:subaccount P

The first is an assignment. The second is a "check" that P is truthy. The third is an instruction to move 100 to account:subaccount. The problem is that the grammar I've written thinks the third line is just an assignment with a missing equal sign. I can't see why.

program: (stmt NEWLINE)+;

stmt: require | entry;

require: 'require' filtrex;
entry: (CREDIT | DEBIT) JOURNAL filtrex (IF filtrex)? (LPARENHASH EXTID RPAREN)?;
assign: ID EQ filtrex;

filtrex: math;

math
   :  math (TIMES | DIV)  math
   |  math (PLUS | MINUS) math
   |  LPAREN math RPAREN
   |  (PLUS | MINUS)* atom
   ;

atom: NUMBER
   | ID
   ;

NUMBER
   : ('0' .. '9') + ('.' ('0' .. '9') +)?
   ;

fragment SIGN
   : ('+' | '-')
   ;

ID: [a-zA-Z]+[0-9a-zA-Z]*;
EQ: '=';
JOURNAL: [a-zA-Z:]+;
EXTID: [a-zA-Z0-9-]+;
COLON: ':';
CREDIT: 'credit';
DEBIT: 'debit';
IF: 'if';
NEWLINE : [
];
NUM     : [0-9.]+;
LPAREN: '(';
RPAREN: ')';
LPARENHASH: '(#';
PLUS: '+';
MINUS: '-';
TIMES: '*';
DIV: '/' ;
POINT: '.';
WS: [ 
	] + -> skip;

UPDATE Thanks to the suggestions below I have something that seems to work properly. Now to the implementation of the logic...

grammar Txl;

// High level language
program: stmt (NEWLINE stmt)* NEWLINE? EOF;

stmt: require | entry | assignment;

require: 'require' expr;
entry: (CREDIT | DEBIT) journal expr (IF expr)? (LPAREN 'id:' EXTID RPAREN)?;
assignment: IDENT ASSIGN expr;

journal: IDENT COLON IDENT;

expr: expr MULT expr
    | expr DIV expr
    | expr PLUS expr
    | expr MINUS expr
    | expr MOD expr
    | expr POW expr
    | MINUS expr
    | expr AND expr
    | expr OR expr
    | NOT expr
    | expr EQ expr
    | expr NEQ expr
    | expr LTE expr
    | expr LT expr
    | expr GTE expr
    | expr GT expr
    | expr QUESTION expr COLON expr
    | LPAREN expr RPAREN
    | NUMBER
    | IDENT LPAREN args RPAREN
    | IDENT
    ;

fnArg: expr | journal;

args: fnArg
    | fnArg COMMA fnArg
    |
    ;

// Reserved words
CREDIT: 'credit';
DEBIT: 'debit';
IF: 'if';
REQUIRE: 'require';

// Operators
MULT: '*';
DIV: '/';
MINUS: '-';
PLUS: '+';
POW: '^';
MOD: '%';
LPAREN: '(';
RPAREN: ')';
LBRACE: '[';
RBRACE: ']';
COMMA: ',';
EQ: '==';
NEQ: '!=';
GTE: '>=';
LTE: '<=';
GT: '>';
LT: '<';
ASSIGN: '=';
QUESTION: '?';
COLON: ':';
AND: 'and';
OR: 'or';
NOT: 'not';
HASH: '#';
NEWLINE : [
];
WS: [ 	] + -> skip;

// Entities
NUMBER: ('0' .. '9') + ('.' ('0' .. '9') +)?;
IDENT: [a-zA-Z]+[0-9a-zA-Z]*;
EXTID: [a-zA-Z0-9-]+;

user_50476 · Answer

That is because the input credit is not being matched by your CREDIT rule, but by the ID rule. The lexer always tries to match as many characters as possible. So, the input credit can be matched by: ID, JOURNAL, EXTID and CREDIT. Whenever it happens that multiple rules can match the same characters, the one defined first "wins" (ID in this case). The lexer does not "listen" to what the parser is trying to match, it operates independently from the parser.

Note that the EXTID also causes the input - to be matched by it, causing the MINUS rule to never be matched.

The solution: place your keywords before the ID rule inside the grammar:

CREDIT     : 'credit';
DEBIT      : 'debit';
REQUIRE    : 'require';
ID         : [a-zA-Z]+ [0-9a-zA-Z]*;

And, if possible, I'd also remove the JOURNAL and EXTID lexer rules and try to "promote" them to parser rules:

journal
 : ID COLON ID
 ;

extid
 : ID (MINUS ID)*
 ;

NUMBER and NUM can also match the same, while NUM also matches input like 1........2.......22222.... I'd remove the NUM rule and only keep NUMBER.

Remove the part from WS: [ ] + -> skip; since these are already matched by your NEWLINE rule.

By doing (stmt NEWLINE)+, every stmt must end with a new line (also the last one). This could be a better solution: stmt (NEWLINE stmt)* NEWLINE?.

The grammar could look like this:

program
 : stmt (NEWLINE stmt)* NEWLINE? EOF
 ;

stmt
 : require
 | entry
 | assign
 ;

require
 : REQUIRE filtrex
 ;

entry
 : (CREDIT | DEBIT) journal filtrex (IF filtrex)? (LPARENHASH extid RPAREN)?
 ;

assign
 : ID EQ filtrex
 ;

journal
 : ID COLON ID
 ;

extid
 : ID (MINUS ID)*
 ;

filtrex
 : math
 ;

math
 : math (TIMES | DIV)  math
 | math (PLUS | MINUS) math
 | LPAREN math RPAREN
 | (PLUS | MINUS)* atom
 ;

atom
 : NUMBER
 | ID
 ;

NUMBER     : [0-9]+ ('.' [0-9]+)?;
CREDIT     : 'credit';
DEBIT      : 'debit';
REQUIRE    : 'require';
IF         : 'if';
ID         : [a-zA-Z]+ [0-9a-zA-Z]*;
EQ         : '=';
COLON      : ':';
NEWLINE    : [
];
LPAREN     : '(';
RPAREN     : ')';
LPARENHASH : '(#';
PLUS       : '+';
MINUS      : '-';
TIMES      : '*';
DIV        : '/' ;
POINT      : '.';
WS         : [ 	] + -> skip;

which will parse your example input like this:

Simple ANTLR grammar not disambiguating identifiers and literals

1 Answers