How does Scala parser combinators handle whitespace?

Viewed 132

I want to parse a document treating each line as a distinct token. A toy example might look something like this:

     August Pizza Sales
        2021-08-21

     22
     19
     53

     P.S. I <3 

Lines might have arbitrary whitespace before and after the text. (My real case is more complicated, and might have different types of data sections, headers, etc. of arbitrary length.)

Here's a basic attempt:

import java.time.LocalDate
import scala.util.parsing.combinator.RegexParsers

case class Document(
    title: String,
    date:  LocalDate,
    data:  Seq[Integer],

)

class LineParser extends RegexParsers {
    private val eol = sys.props("line.separator")

    def text:     Parser[String]       = ".*".r     ^^ { _.toString }
    def date:     Parser[LocalDate]    = ".*".r     ^^ { LocalDate.parse(_) }
    def blank:    Parser[String]       = "^\\s*$".r ^^ { _.toString }
    def datum:    Parser[Integer]      = "[0-9]+".r ^^ { _.toInt }
    def data:     Parser[Seq[Integer]] = repsep(datum, eol)
    def document: Parser[Document]     = text ~ date ~ blank ~ data ~ blank ~ text ^^ {
        case title ~ date ~ _ ~ data ~ _ ~ footer =>
            Document(title=title, date=date, data=data, footer=footer)
    }
}

When I run this, I get Document(Pizza Sales,2021-08-21,List(23),19).

There are many things I don't understand here ...

Tactical Questions

  • What changes do I need to make to correctly parse the document?
  • If I change my definition of blank to "^\\s*$".r (which is what should actually match a blank line, I think) then everything breaks. Why?

Strategic Questions

  • What's really going on with newlines here? The text parser does not match across newlines, so it seems they're being treated specially (i.e., even if I try to match text to the entire document, it only gets Pizza Sales.)
  • What is the impact of setting skipWhitespace?
  • What is the impact of setting val whiteSpace = ... (in combination with skipWhitespace)?
  • [SUPER BONUS QUESTION] What the heck does ~> mean?

Philosophical Questions

Since I already know how to tokenize my input (i.e., split it on newlines), is RegexParsers (or Scala parser combinators in general) the right tool for this job?

0 Answers
Related