Custom parser for file similar to json python

Viewed 593

I'm attempting to create a parser to translate a "custom" file into JSON so I can more easily manipulate its contents (For argument's sake, call the "custom" formate a .qwerty).

I've already created a Lexer which breaks down the file into individual lexemes (tokens) which structure is [token_type, token_value]. Now I am struggling to parse the lexemes into their correct dictionaries, as it is difficult to insert data into a sub-sub-dictionary since Keys aren't constant. As well as insert data into arrays stored in dictionaries.

It should be noted I am attempting to sequentially parse tokens into an actual python json object then dump the json object.

An example of the file can be seen below, along with what the end result is meant to resemble.

FILE: ABC.querty

Dict_abc_1{

    Dict_abc_2{
        HeaderGUID="";
        Version_TPI="999";
        EncryptionType="0";
    }

    Dict_abc_3{
        FamilyName="John Doe";
    }

    Dict_abc_4{
        Array_abc{
            {TimeStamp="2018-11-07 01:00:00"; otherinfo="";}
            {TimeStamp="2018-11-07 01:00:00"; otherinfo="";}
            {TimeStamp="2018-11-07 01:00:00"; otherinfo="";}
            {TimeStamp="2018-11-07 02:53:57"; otherinfo="";}
            {TimeStamp="2018-11-07 02:53:57"; otherinfo="";}
        }

        Dict_abc_5{
            LastContact="2018-11-08 01:00:00";
            BatteryStatus=99;
            BUStatus=PowerOn;
            LastCallTime="2018-11-08 01:12:46";
            LastSuccessPoll="2018-11-08 01:12:46";
            CallResult=Successful;
        }
    }
}
Code=999999;

FILE: ABC.json

{  
    "Dict_abc_1":{
        "Dict_abc_2":{
            "HeaderGUID":"",
            "Version_TPI":"999",
            "EncryptionType":"0"
        },

        "Dict_abc_3":{
            "FamilyName":"John Doe"
        },

        "Dict_abc_4":{
            "Array_abc":[
                {"TimeStamp":"2018-11-07 01:00:00", "otherinfo":""},
                {"TimeStamp":"2018-11-07 01:00:00", "otherinfo":""},
                {"TimeStamp":"2018-11-07 01:00:00", "otherinfo":""},
                {"TimeStamp":"2018-11-07 02:53:57", "otherinfo":""},
                {"TimeStamp":"2018-11-07 02:53:57", "otherinfo":""}
            ],

            "Dict_abc_5":{
                "LastContact":"2018-11-08 01:00:00",
                "BatteryStatus":99,
                "BUStatus":"PowerOn",
                "LastCallTime":"2018-11-08 01:12:46",
                "LastSuccessPoll":"2018-11-08 01:12:46",
                "CallResult":"Successful"
            }
        }
    },
    "Code":999999
}

Additional token information, Token types can either be (with possible values)

  • IDENTIFIER contain the name of the variable identifier
  • VARIABLE containing actual data belonging to the parent IDENTIFIER
  • OPERATOR equal "="
  • OPEN_BRACKET equal "{"
  • CLOSE_BRACKET equal "}"

An example of ABC.querty's lexemes can be seen HERE

fundamental logical extract of main.py

def main():
    content = open_file(file_name) ## read file

    lexer = Lexer(content) ## create lexer class
    tokens = lexer.tokenize() ## create lexems as seen in pastebin

    parser = Parser(tokens).parse() ## create parser class given tokens 

    print(json.dumps(parser, sort_keys=True,indent=4, separators=(',', ': ')))

parser.py

import re

class Parser(object):
    def __init__(self, tokens):
        self.tokens = tokens
        self.token_index = 0
        self.json_object = {}
        self.current_object = {}
        self.path = [self.json_object]

    def parse(self):
        while self.token_index < len(self.tokens):

            token = self.getToken()
            token_type = token[0]
            token_value = token[1]

            print("%s \t %s" % (token_type, token_value))

            if token_type in "IDENTIFIER":
                self.increment()
                identifier_type = self.getToken()
                if identifier_type[0] in "OPEN_BRACKET":
                    identifier_two_type = self.getToken(1)
                    if identifier_two_type[0] in ["OPERATOR","IDENTIFIER"]:
                        ## make dict in current dict 
                        pass

                    elif identifier_two_type[0] in "OPEN_BRACKET":
                        ## make array in current dict 
                        pass

                elif identifier_type[0] in "OPERATOR":
                    ## insert data into current dict
                    pass


            if token_type in "CLOSE_BRACKET":
                identifier_type = self.getToken()
                if "OPEN_BRACKET" in identifier_type[0]:
                    #still in array of current dict
                    pass
                elif "IDENTIFIER" in identifier_type[0]:
                    self.changeDirectory()

                else:
                    #end script
                    pass

            self.increment()
        print(self.path)
        return self.json_object


    def changeDirectory(self):
        if len(self.path) > 0:
            self.path = self.path.pop()
            self.current_object = -1

    def increment(self):
        if self.token_index < len(self.tokens):
            self.token_index+=1

    def getToken(self, x=0):
        return self.tokens[self.token_index+x]

Additional parse information, Currently, I was trying to store the current dictionary in a path array to allow me to insert into dictionaries and arrays within dictionaries.

Any suggestions or solutions are very much appreciated,

Thanks.

1 Answers

Last time I solved this problem I find out that finite-state machine is very helpful. I want to recommend the way after you have tokens but I don't know how it's called in english. The principle is: you go through tokens and add one by one on stack. After adding on stack you are checking stack for some rules. Like you combine primitive tokens into expressions that might be a part of more complex expressions.

For example "FamilyName":"John Doe". Tokens are "FamilyName", : and "John Doe".

You add first token on stack. stack = ["FamilyName"]. Rule 1: str_obj -> E. So you create Expression(type='str', value="FamilyName") and stack is now stack = [Expression].

Then you add next token. stack = [Expression, ':']. No rules for ':'. Go next.

stack = [Expression, ':', "FamilyName"]. Again we meet rule 1. So stack becomes stack = [Expression, ':', Expression]. Then we see another rule. Rule 2: E:E -> E. Use it like Expression(type='kv_pair, value=(Expression, Expression)). And stack becomes stack=[Expression].

And if you describes all the rules it will work like that. Hope it helps.

Related