Convert repetitive pattern into JSON file with multiple arrays in Python

user_2528324 0
Viewed 114

I do have a large file of configuration which is saved in a repetitive pattern as shown below example

SLAVE="abc"
    SSI_SENDER="X"
    SSI_RECEIVER="Y"
    COMMENT=""
        SENDER_EXT="Z"
            SENDER_COMMENT="L"
        SENDER_EXT="P"
            SENDER_COMMENT="M"
        RECEIVER_EXT="A"
            RECEIVER_COMMENT=""
        RECEIVER_EXT="B"
            RECEIVER_COMMENT="D"
        SSI_DATA="Q"
            SSI_DATA_COMMENT=""
                MESSAGEID="D"
                    MESSAGE_COMMENT=""
                MESSAGEID="W"
                    MESSAGE_COMMENT=""
        SSI_DATA="V"
            SSI_DATA_COMMENT="" 
                MESSAGEID="K"
                MESSAGE_COMMENT=""

SLAVE="def"
    SSI_SENDER="sd"
    SSI_RECEIVER="PL"
    COMMENT=""
        SENDER_EXT="AB"
            SENDER_COMMENT=""
        SENDER_EXT="CD"
            SENDER_COMMENT=""
        RECEIVER_EXT="KL"
            RECEIVER_COMMENT=""
        RECEIVER_EXT="ML"
            RECEIVER_COMMENT=""
        SSI_DATA="AI"
            SSI_DATA_COMMENT=""
                MESSAGEID="VV"
                    MESSAGE_COMMENT=""
                MESSAGEID="CC"
                     MESSAGE_COMMENT=""

There are few things to keep in mind, there can be many

  • For every loop starting from SLAVE till next loop, We can have many SENDER_EXT, RECEIEVER_EXT, SSI_DATA, and the MESSAGE ID
  • SENDER_ID array in JSON should contain all the SENDER_EXT along with SENDER_COMMENT
  • RECEIVER_ID array in JSON should contain all the RECEIVER_EXT along with RECEIVER_COMMENT
  • SSI_DATA array in JSON contains all the SSI_DATA and which further contains information like MESSAGEID and MESSAGE_COMMENT in its subarray

I am able to get the output in a single JSON file with no multiple arrays when there is only SENDER_EXT, RECEIVER_EXT, SSI_DATA but not when they come more than once

Using the python code

import re
import json

# Read from file or use the dummy data
with open("../data/input/ssi_demo.cfg", "r") as f:
    s = f.read()

field_labels = [
                'SLAVE',
                'SSI_SENDER', 
                'SSI_RECEIVER', 
                'COMMENT',
                'SENDER_EXT',
                'SENDER_COMMENT',
                'RECEIVER_EXT',
                'RECEIVER_COMMENT',
                'SSI_DATA',
                'SSI_DATA_COMMENT',
                'MESSAGEID'
                ]

# Define regex pattern and compile for speed
pat = '="(.*)"\n\s*'.join(field_labels) + '="(.*)"'
#print(pat)
pat = re.compile(pat)
#print(pat)
data=pat.findall(s)

# Prepare a list of dicts: each dict for a single block of data
d = [dict((k,v) for k,v in zip(field_labels, field_values)) for field_values in data]
text = json.dumps({'data': d}, indent=2)
#print(text)
# Write to a json file
with open('../data/output/ssi_output_demo.json', 'w') as f:
    f.write(text)

print("JSON Created at data/output")

Expected Output

{
  "data": [
    {
      "SLAVE": "abc",
      "SSI_SENDER": "X",
      "SSI_RECEIVER": "Y",
      "COMMENT": "",
      "SENDER_ID":[
                {"SENDER_EXT": "Z","SENDER_COMMENT": "L"},
                 {"SENDER_EXT": "P","SENDER_COMMENT": "M"},
                ],
      "RECEIVER_ID":[
                   {"RECEIVER_EXT": "A","RECEIVER_COMMENT": ""},
                   {"RECEIVER_EXT": "B","RECEIVER_COMMENT": "D"}
                ],

      "SSI_DATA": [
                    "Q":{"MESSAGEID": "D",MESSAGE_COMMENT=""},
                    "V":{"MESSAGEID": "K",MESSAGE_COMMENT=""}
                    ]
    
    },
python-re python json regex
1 Answers

The classic way to deal with this kind of parsing is not via a regex, but rather treating it as a context-free grammar. You can then use parsers such as PyParsing or Lark to parse it. Here's an example of how to do it with Lark:

from lark import Lark
import lark 

grammar = """start: slave +
slave: "SLAVE" "=" ESCAPED_STRING (ssi_sender|ssi_receiver|comment)* 
ssi_sender: "SSI_SENDER" "=" ESCAPED_STRING 
ssi_receiver: "SSI_RECEIVER" "=" ESCAPED_STRING 
comment: "COMMENT" "=" ESCAPED_STRING (sender_ext|receiver_ext|ssi_data)+
receiver_ext: "RECEIVER_EXT" "=" ESCAPED_STRING receiver_comment
receiver_comment: "RECEIVER_COMMENT" "=" ESCAPED_STRING
sender_ext: "SENDER_EXT" "=" ESCAPED_STRING sender_comment
sender_comment: "SENDER_COMMENT" "=" ESCAPED_STRING
ssi_data: "SSI_DATA" "=" ESCAPED_STRING ssi_data_comment
ssi_data_comment: "SSI_DATA_COMMENT" "=" ESCAPED_STRING message+ 
message: "MESSAGEID" "=" ESCAPED_STRING message_comment
message_comment: "MESSAGE_COMMENT" "=" ESCAPED_STRING   

%import common.ESCAPED_STRING
%import common.WS
%import common.NEWLINE
%ignore WS 
%ignore NEWLINE """


p = Lark(grammar, parser='lalr', debug=True)
res = p.parse(text)

def tree_to_dict(tree):
    if isinstance(tree, lark.Tree):
        if len(tree.children) > 1:
            return {tree.data: [tree_to_dict(c) for c in tree.children]} 
        else:
            return {tree.data: tree_to_dict(tree.children[0])} 
    return tree.value[1:-1]
tree_to_dict(res)

The output of this parser, after being converted to a dictionary, is given below. You can easily manipulate it to a slightly different structure if you'd like to.

{'start': [{'slave': ['abc',
    {'ssi_sender': 'X'},
    {'ssi_receiver': 'Y'},
    {'comment': ['',
      {'sender_ext': ['Z', {'sender_comment': 'L'}]},
      {'sender_ext': ['P', {'sender_comment': 'M'}]},
      {'receiver_ext': ['A', {'receiver_comment': ''}]},
      {'receiver_ext': ['B', {'receiver_comment': 'D'}]},
      {'ssi_data': ['Q',
        {'ssi_data_comment': ['',
          {'message': ['D', {'message_comment': ''}]},
          {'message': ['W', {'message_comment': ''}]}]}]},
      {'ssi_data': ['V',
        {'ssi_data_comment': ['',
          {'message': ['K', {'message_comment': ''}]}]}]}]}]},
  {'slave': ['def',
    {'ssi_sender': 'sd'},
    {'ssi_receiver': 'PL'},
    {'comment': ['',
      {'sender_ext': ['AB', {'sender_comment': ''}]},
      {'sender_ext': ['CD', {'sender_comment': ''}]},
      {'receiver_ext': ['KL', {'receiver_comment': ''}]},
      {'receiver_ext': ['ML', {'receiver_comment': ''}]},
      {'ssi_data': ['AI',
        {'ssi_data_comment': ['',
          {'message': ['VV', {'message_comment': ''}]},
          {'message': ['CC', {'message_comment': ''}]}]}]}]}]}]}
user_11055600
Related