ANTLR: handle emoji characters

user_2460378 0
Viewed 163

I'm using following EmojiLexer.g4 together with ANTLR 4.7.2

lexer grammar EmojiLexer;

String
    : '"'   StringChar*  '"'
    ;

fragment StringChar
    : ~["\r\n]
    ;

Punctuation
    : ( '['
      | ':'
      | ']'
      | ';'
      | '@'
      )
    ;

fragment IdentifierNonDigit
    : [a-zA-Z_$\u0080-\uffff]
    ;

Identifier
    : IdentifierNonDigit+
    ;

Whitespace
    : [ \t]+
      -> skip
    ;

Newline
    : ( '\r' '\n'?
      | '\n'
      )
      -> skip
    ;

and following Java code to lex a string:

public class EmojiTest {

public static void main(String[] args) {
    final String string = "[foo bar:bazz];\n"
                          + "\n"
                          + "@\"emojis break it: \uD83D\uDE31\";\n"
                          + "\n"
                          + "[foo bar:bazz];\n";
    final CharStream charStream = CharStreams.fromString(string);
    final EmojiLexer lexer = new EmojiLexer(charStream);
    while (true) {
        final Token token = lexer.nextToken();
        final int type = token.getType();
        if (type < 0) {
            break;
        }

        final int startIndex = token.getStartIndex();
        final int stopIndex = token.getStopIndex() + 1;
        System.out.println(startIndex + "-" + stopIndex + ": " + type + ": " + escape(string.substring(startIndex, stopIndex)));
    }
}

private static String escape(String s) {
    final StringBuilder buffer = new StringBuilder();
    for (int i = 0; i < s.length(); i++) {
        final char chr = s.charAt(i);
        if (chr >= 0x20 && chr < 0x7f) {
            buffer.append(chr);
        }
        else {
            buffer.append("\\u");
            final String hex = Integer.toHexString(chr);
            for (int j = hex.length(); j < 4; j++) {
                buffer.append('0');
            }
            buffer.append(hex);
        }
    }
    return buffer.toString();
}

}

but somehow the emoji double-character confuses ANTLR, because the result is:

0-1: 2: [
1-4: 3: foo
5-8: 3: bar
8-9: 2: :
9-13: 3: bazz
13-14: 2: ]
14-15: 2: ;
17-18: 2: @
18-38: 1: "emojis break it: \ud83d\ude31
38-39: 2: "
41-42: 2: \u000a
42-45: 3: [fo
46-49: 3:  ba
49-50: 2: r
50-54: 3: :baz
54-55: 2: z
55-56: 2: ]

Is this a bug in my code or in ANTLR?

antlr4 antlr unicode java
2 Answers

You used a surrogate pair to specify the emoji. I'm not sure this is still supported. Instead use the full UTF-32 specification: \u{1f631} for

If you want to capture all emojis, not just this single one you can even use:

EMOJI : [\p{Emoji}];

which uses the Unicode character class for emojis.

See also: https://github.com/antlr/antlr4/blob/master/doc/unicode.md#unicode-code-points-in-lexer-grammars

By exchanging the surrogate by the UTF-32 character I get this list of tokens:

[@0,0:0='[',<2>,1:0]
[@1,1:3='foo',<3>,1:1]
[@2,5:7='bar',<3>,1:5]
[@3,8:8=':',<2>,1:8]
[@4,9:12='bazz',<3>,1:9]
[@5,13:13=']',<2>,1:13]
[@6,14:14=';',<2>,1:14]
[@7,17:17='@',<2>,3:0]
[@8,18:37='"emojis break it: "',<1>,3:1]
[@9,38:38=';',<2>,3:21]
[@10,41:41='[',<2>,5:0]
[@11,42:44='foo',<3>,5:1]
[@12,46:48='bar',<3>,5:5]
[@13,49:49=':',<2>,5:8]
[@14,50:53='bazz',<3>,5:9]
[@15,54:54=']',<2>,5:13]
[@16,55:55=';',<2>,5:14]
[@17,57:56='<EOF>',<-1>,6:0]

This is the input I used:

[foo bar:bazz];

@"emojis break it: ";

[foo bar:bazz];

I have to admit that I tested it with my ANLTR4 extension, which uses Typescript instead of Java, but I believe that shouldn't matter.

user_11371740

Problem is that ANTLR is codepoint-aware and the returned indices are codepoint-indices. So the application code needs to be adapted to handle codepoints as well.

user_24603780
Related