ANTLR4 lexer getTokens() method returns only the <EOF> token

user_14645096 0
Viewed 105

Basically, I have the following code:


assert lexer != null;
CommonTokenStream commonTokenStream = new CommonTokenStream(lexer);commonTokenStream.fill();

List<Token> lst = commonTokenStream.getTokens();
FileWriter f = new FileWriter(output);

            for (Token t : lst) {
                f.write(t.toString());
                System.out.println(t.getText());
            }
f.close();

But, the file only gets the <EOF> token written to it, am I missing something? I'm certain there's more tokens in the file.

antlr4 lexer java
1 Answers

When I copy-paste your code, I cannot reproduce it.

This is the test grammar:

grammar T;

parse
 : ANY*? EOF
 ;

ANY
 : .
 ;

and Java code:

TLexer lexer = new TLexer(CharStreams.fromString("mu"));
CommonTokenStream commonTokenStream = new CommonTokenStream(lexer);
commonTokenStream.fill();

List<Token> lst = commonTokenStream.getTokens();

for (Token t : lst) {
  System.out.println(t.getText());
}

produces the following output:

m
u
<EOF>

Most likely, you already consumed all tokens using an instance of a parser. Do a lexer.reset(); before you create a CommonTokenStream:

lexer.reset();
CommonTokenStream commonTokenStream = new CommonTokenStream(lexer);
...
user_504760
Related