Parsing key / value subparameters

user_3724999 0
Viewed 58

I'm a bit clueless as to how I can parse (more or less) "free form" parameter lists, suppose the syntax allows for

PARM=(VAL1, 'VAL2', VAL3, KEY4=VAL4, KEY5=VAL5(XYZ), PARM=ABC, SOMETHING=ELSE)

I have managed to basically parse combos of positional and key/value parameters, but as soon as I hit a lexer token like PARM= the parser bails out with a "mismatched input", and I can't specifically allow for or expect anything because these parameters passed to a function are completely arbitrary.

So I'd think I'll need to switch to a specific lexer mode but right now I can't see how I would properly switch back to "normal" mode, the delimiters are PARM=( on the left and the closing ) on the right, but as the "data" itself can contain (pairs of) brackets how would I identify the correct closing paren so I don't prematurely end the lexer mode?

TIA - Alex

Edit 1:

Minimal grammar showing the issue with keywords being used where they shouldn't, as this is part of a complex grammar I can't change the order of tokens to put ID in front of everything else, for example, as it would catch too much. So I don't see how this can work short of breaking out into a different lexer mode.

lexer grammar ParmLexer;

  SPACE     :   [ \t\r\n]+    -> channel(HIDDEN) ;
  COMMA     :   ',' ;
  EQUALS    :   '=' ;
  LPAREN    :   '(' ;
  RPAREN    :   ')' ;

  PARM      :   'PARM=' ;

  ID        :   ID_LITERAL ;
  fragment ID_LITERAL   :   [A-Za-z]+ ;

.

parser grammar ParmParser;

options { tokenVocab=ParmLexer; }

parms :     PARM LPAREN parm+ RPAREN    ;
parm :      (pkey=ID EQUALS)? pval=ID COMMA?    ;

Input:

PARM=( TEST, KEY=VAL, PARM=X)

Results in

line 1:22 extraneous input 'PARM=' expecting {')', ID}
antlr4
2 Answers

So I'd think I'll need to switch to a specific lexer mode but right now I can't see how I would properly switch back to "normal" mode

Instead of switching to modes (with -> mode(...)), you can push your "special" mode on a stack (with -> pushMode(...)) and then when encountering a ) you pop a mode from the stack. That way, you can have multiple nested lists (..(..(..).)..). A quick demo:

lexer grammar ParmLexer;

SPACE  : [ \t\r\n]+ -> channel(HIDDEN);
EQUALS : '=' ;
LPAREN : '(' -> pushMode(InList);
PARM   : 'PARM';
ID     : [A-Za-z] [A-Za-z0-9]*;

mode InList;

 LST_LPAREN : '(' -> type(LPAREN), pushMode(InList);
 RPAREN     : ')' -> popMode;
 COMMA      : ',';
 LST_EQUALS : '=' -> type(EQUALS);
 STRING     : '\'' ~['\r\n]* '\'';
 LST_ID     : [A-Za-z] [A-Za-z0-9]* -> type(ID);
 LST_SPACE  : [ \t\r\n]+ -> channel(HIDDEN);

and:

parser grammar ParmParser;

options { tokenVocab=ParmLexer; }

parse
 : PARM EQUALS list EOF
 ;

list
 : LPAREN ( value ( COMMA value )* )? RPAREN
 ;

value
 : ID
 | STRING
 | key_value
 | ID list
 ;

key_value
 : ID EQUALS value
 ;

which will parse your example input PARM=(VAL1, 'VAL2', VAL3, KEY4=VAL4, KEY5=VAL5(XYZ), PARM=ABC, SOMETHING=ELSE) like this:

enter image description here

user_504760

You don't have a rule (alternative) that recognizes a PARM token in your parm rule.

Bart has provided an answer using Lexer modes (and assuming that LPAREN and RPAREN always control those modes), but you can also just set up a parser rule that matches all of your keywords:

lexer grammar ParmLexer
    ;

SPACE:  [ \t\r\n]+ -> channel(HIDDEN);
COMMA:  ',';
EQUALS: '=';
LPAREN: '(';
RPAREN: ')';

PARM: 'PARM';
KW1:  'KW1';
KW2:  'KW2';

ID:                  ID_LITERAL;
fragment ID_LITERAL: [A-Za-z]+;

parser grammar ParmParser
    ;

options {
    tokenVocab = ParmLexer;
}

parms: PARM EQUALS LPAREN parm (COMMA parm)* RPAREN;
parm:  ((pkey = ID | kwid = kw) EQUALS)? pval = ID;
kw:    PARM | KW1 | KW2;

input

"PARM=( TEST, KEY=VAL, KW2=v2, PARM=X)"

yields:

(parms PARM = ( (parm TEST) , (parm KEY = VAL) , (parm (kw KW2) = v) , (parm (kw PARM) = X) ))
user_737640
Related