golang participle panic with "panic: unexpected token XXX"

user_4785390 0
Viewed 376

I am learning participle, a parser based on golang. After I went through the tutorial, I decided to create a simple Array parser, here's my construction.

First, I need an AST to represent this array. for the simplicity purpose, I only parse Array of string without quotation mark. ({value1, value_two, value_3_here}, e.t.)

type SimpleArray struct {
    Arr []Values `"{" @@* "}"`
}

type Values struct {
    Str string `@Ident`
}

and now I create a Lexer for the input string

var Lexer = stateful.MustSimple([]stateful.Rule{
    {`Ident`, `[a-zA-Z][a-zA-Z_\d]*`, nil},
    {`punct`, `[}{,]`, nil},
    {`whitespace`, `\s+`, nil},
})

and I have my main function below, with a simple test case s := "{key, value}"

func main() {
    var parser = participle.MustBuild(&SimpleArray{},
        participle.Lexer(Lexer),
    )
    b := &SimpleArray{}
    s := "{key, value}"
    err := parser.ParseString("", s, b)
    if err != nil {panic(err)}
    repr.Println(b, repr.Indent("  "), repr.OmitEmpty(true))
}

But I got a panic.

panic: 1:2: unexpected token "key"

I think the @Ident in Values struct will catch the "key", but it isn't, why is it? and how can I fix this?

Thanks!

go lexer parsing
1 Answers

There are two problems that I see, the first being a rule definition syntax error, the second being an error with the rules themselves.

The parser is not able to parse anything, because the parser still needs to get a { token to begin the first rule SimpleArray. The reason why the first { token is not being passed through by the lexer is because the name of the rule punct should be capitalized Punct. Tokens in a lowercase rule (like whitespace) are read by the lexer, but not passed to the parser.

Additionally, there is another problem that is not letting you parse the commas. You need to modify the rule to allow for additional items which have the separating comma.

Change the following:

type SimpleArray struct {
    Arr []Values `"{" @@* "}"`
}

to:

type SimpleArray struct {
    Arr []Values `"{" (@@ ("," @@)*)? "}"`
}

An explanation of the new rule:

  • "{" to parse the {
  • (...)? to make the list optional (if this is what you actually want, if not remove this part)
  • @@ ("," @@)* to recursively parse one or more items, separated by a comma
  • "}" to parse the }
user_51418750
Related