Scala Packrat Parsers, error messages and referential transparency

Viewed 197

I'm trying to parse boolean expressions into an Expr tree using Scala's Packrat parsers from scala-parser-combinators.

  sealed trait Expr
  case class Term(term: String) extends Expr
  case class And(x: Expr, y: Expr) extends Expr
  case class Or(x: Expr, y: Expr) extends Expr
aaa and bbb          --> And(Term(aaa),Term(bbb))
aaa and bbb or ccc   --> Or(And(Term(aaa),Term(bbb)),Term(ccc))
aaa and (bbb or ccc) --> And(Term(aaa),Or(Term(bbb),Term(ccc)))

This grammar seems to work perfectly fine:

object Parsing extends RegexParsers with PackratParsers {

  override val skipWhitespace = false

  val validIdentifiers = List("aaa", "bbb", "ccc", "ddd")

  lazy val term: PackratParser[Term] = """\s*""".r ~> """\w+""".r flatMap { identifier =>
    if (validIdentifiers.contains(identifier))
      success(Term(identifier))
    else
      err(s"expected one of: $validIdentifiers")
  }

  lazy val and: PackratParser[And] =
    expr ~ """\s+and\s+""".r ~ (term | parensExpr) ^^ { case e1 ~ _ ~ e2 => And(e1, e2) }

  lazy val or: PackratParser[Or] =
    expr ~ """\s+or\s+""".r ~ (term | parensExpr) ^^ { case e1 ~ _ ~ e2 => Or(e1, e2) }

  lazy val parensExpr: PackratParser[Expr] = """\s*\(""".r ~> expr <~ """\s*\)""".r

  lazy val expr: PackratParser[Expr] =
    term ||| and ||| or ||| parensExpr

  lazy val root: PackratParser[Expr] =
    phrase(expr)

  def parseExpr(input: String): ParseResult[Expr] =
    parse(root, new PackratReader(new CharSequenceReader(input)))
}

But error messages are sometimes.... bad. If the parser finds an invalid identifier on the left-hand side of an and, it'll correctly tell us so.

println(parseExpr("invalidIdentifier and aaa"))


[1.18] error: expected one of: List(aaa, bbb, ccc, ddd)
invalidIdentifier and aaa
                 ^

But if it finds an invalid identifier on the right-hand side of an and, it'll give us this misleading error message.

println(parseExpr("aaa and invalidIdentifier"))

[1.4] failure: end of input expected
aaa and invalidIdentifier
   ^

I'm pretty sure this happens because expr will try all 4 options: and/or/parensExpr will fail, but term will succeed with just Term("aaa").

Then, root's phrase will kick in and check whether there's any input left to consume, and fail because there is: " and invalidIdentifier".


So I thought, I'll push phrase one level down. In other words, I changed this:

  lazy val expr: PackratParser[Expr] =
    term ||| and ||| or ||| parensExpr

  lazy val root: PackratParser[Expr] =
    phrase(expr)

Into this:

  lazy val expr: PackratParser[Expr] =
    term ||| and ||| or ||| parensExpr

  lazy val root: PackratParser[Expr] =
    phrase(term) ||| phrase(and) ||| phrase(or) ||| phrase(parensExpr)

Now, all 4 options should fail, but we should see and's error message because and consumed more input than the other 3 options

I'm now getting better error messages BUT, to my surprise, some previously valid inputs are now invalid!

println(parseExpr("aaa or bbb"))

[1.4] failure: string matching regex '\s+and\s+' expected but ' ' found
aaa or bbb
   ^
println(parseExpr("aaa and bbb or ccc"))

[1.12] failure: end of input expected
aaa and bbb or ccc
           ^

I don't understand why.


In fact, even just a simpler, trivial change such as this:

  // before
  lazy val root: PackratParser[Expr] =
    phrase(expr)

  // after
  lazy val root: PackratParser[Expr] =
    phrase(term ||| and ||| or ||| parensExpr)

... breaks previously valid inputs.

How come? Shouldn't these two definitions of root be equivalent? Are these parsers not referentially transparent?

More importantly, how should I go about fixing this?

1 Answers

I'm having the following errors with the exact code you posted, I believe these are the expected error messages?:

@ println(Parsing.parseExpr("invalidIdentifier and aaa"))
[1.18] error: expected one of: List(aaa, bbb, ccc, ddd)

invalidIdentifier and aaa
                 ^


@ println(Parsing.parseExpr("aaa and invalidIdentifier"))
[1.26] error: expected one of: List(aaa, bbb, ccc, ddd)

aaa and invalidIdentifier
                         ^

I'm using ammonite repl with ivy imported Scala Parser Combinators library:

import $ivy.`org.scala-lang.modules::scala-parser-combinators:1.1.2`

Maybe a library version issue?

Related