how to do multi-line parsing with python's sly.Parser class?

user_16808372 0
Viewed 51

I am trying to create my own programming language with a library called sly. I have created a lexer to tokenize my program but I am stuck on getting the parser to successfully parse multiple instructions. When I didn't take this into account and tried to make a program to test the parse tree, an it gave me an error saying that a bunch of tokens were invalid. The temporary solution I came up with was to use the following parse statement:

@_("expr \n expr")
def expr(self, t):
    return [t.expr0, t.expr1]
    # I have also tried:
    #  return t.expr1, t.expr2

This leads to an annoying bug were the parse tree would nest the statements to much.

[
 {
  "some statement",
  ...,
  "inner": [
   {
    "another statment",
     ...,
     "inner": [
      {
       "another statment",
       ...,
       "inner": []
      }
     ]
    }
  ]
 }
]

I want the parse tree to be flat so you only get the necessary nesting. Like the parse tree below.

[
 {
  "another statment",
  ...,
  "inner": []
 },
 {
  "another statment",
  ...,
  "inner": []
 },
 {
  "another statment",
  ...,
  "inner": []
 }
]

I was thinking about keeping the parse tree ugly and then reformatting it, but depending on the size of the tree it could make programs extremely slow.

The language is also part of a project I have, github repo linked here:

for all important files: https://github.com/0x32767/0x102-discord-bot/tree/star2py/jpl4py/lang

the parsing class: https://github.com/0x32767/0x102-discord-bot/blob/a268fe9a63f87a9ebf39088cff13ee9a2edab931/jpl4py/lang/jpl.py#L81

the parse tree that is genorated: https://github.com/0x32767/0x102-discord-bot/blob/star2py/jpl4py/lang/hWorld.jpl.out.json

the quick-fix line: https://github.com/0x32767/0x102-discord-bot/blob/a268fe9a63f87a9ebf39088cff13ee9a2edab931/jpl4py/lang/jpl.py#L217

Thank you for reading

sly python json
2 Answers

summery

I managed to solve this problem by adding newlines to the ignore list.

More detail

Every sly.Lexer class can take an ignore class variable. I added newlines so \n into this variable, so now it looks like this: ignore = "\t \n" so now my lexer class looks like:


class MyLexer(sly.Lexer):
    ignore = "\t \n" # this is the edit that I made

    # the rest of the MyLexer class

Notice

This could require semicolons ; to separate individual statements, but this is a matter of personal preference.

p.s I apologise if answering your own question here is a good practice or no, but I don't want anyone else to be having this problem.

user_168083720

I'm still a bit of a newbie at Sly, having started with Ply, but what you need is a rule that encompasses multiple statements and the easiest way to do it is via recursion. So you would wind up with a base statement like the following:

  expr_set : expr_set expr NEWLINE
           | expr NEWLINE

expr_set always exports a set of results. If you hit the second case, that set is that single expression. If it's the first one, then it's the existing set with the new expression appended. So, you wind up with the following code:

@_("expr_set expr \n")
def expr_set(self, t):
    myset = t.expr_set
    myset.append(t.expr)
    return myset

@_("expr \n")
def expr_set(self, t):
    return [t.expr]

Does that make sense? As a side note, that same expr_set can be used for any nested bit of code, such as in a function, or a loop. If newlines are significant for you (as seems to be the case), you may need a 3rd or 4th clause to deal with cases such as a line that's only a newline, and the final statement of the program (or section of commands) not having a newline after it. I asked a similar question here.

Also, another minor pitfall I ran into for early implementation is that you're going to want to not implement the suggested newline routine in the code example, since it will ignore and eat those newlines.

user_9010830
Related