What is the right approach to parse unstructured but somewhat hierarchical data from a file?

user_18811295 0
Viewed 38

I'm learning Python and I want to be able to do the following. As long as I'm still in Item1, I want to find all subsectors and gather the data within the subsectors until I hit the next subsector. Being in Subsector1 the data can be of depth/length 4 or 5 until the next data block is reached. There are no empty lines in the file but I could do a regex on each DataX to figure out what it is and where it belongs. A normal entry includes 4 lines. If it's 5 until the next Data1 I want to concat Data1 + Data2 and go on. This process repeats for all subsectors until Item2 is reached and all subsectors and there data are being extracted. The process ends when there are no more lines to parse.

I'm coming to the conclusion that I probably need to write a line parser with look ahead. Does something in the Python ecosystem exists or is there a simpler approach?

The data comes from a normal text file line by line but the structure I want to extract looks something lime this:

Item1

    SubSector1

    Data1
    Data2 
    Data3
    Data4
    Data5 [if 5 then Data1 belongs to Data2]

    Data1
    Data2
    Data3
    Data4

    ...

    SubSector2

    Data1
    ...
    Data4

    Data1
    ...
    Data5

    SubSector3
    ...

Item2

    SubSector1

    Data1
    ...
    Data4

    SubSector3
    ...

Item3
    ...

In the end I want to create a CSV file that would have the following structure:

0, 1, 2, 3, 4, 5
SubSector1, Item1, Data1 + Data2, Data3, Data4, Data5
SubSector1, Item1, Data1, Data2, Data3, Data4
SubSector2, Item1, Data1 + Data2, Data3, Data4, Data5
SubSector3, Item1, Data1, Data2, Data3, Data4
SubSector1, Item2, Data1, Data2, Data3, Data4
SubSector1, Item2, Data1, Data2, Data3, Data4
SubSector1, Item3, Data1, Data2, Data3, Data4
SubSector1, Item3, Data1 + Data2, Data3, Data4, Data5
...
algorithm parsing python
1 Answers

You used yourself the key word: "depth".

Use a recursive function: when you encounter either a new item or a subsector, you return, providing to caller:

  • Encountered a Data: add value in a Data array. Array is empty at start of a SubSector.
  • Encountered a SubSector: return array of Data read since start of SubSector.
  • Encountered a Item: return array of SubSector, each containing an array of Data.
  • Encountered end of file: Return an array of Items, each containing an array of SubSector, each containing an array of Data. Also, reading process is obviously done.
  • The line/key just read that broke the reading loop: since you don't have "close" markers (like XML), you'll return the "open" marker that you read "unnecessarily" - so caller can call itself recursively with the proper mode. It will also saves you to read ahead / go back in file, which can be a pain with text files if you didn't load them fully into memory.
  • No line/key must be altered until "consumed" by something. If the Data part is reading an Item, then the middle SubSector part must NOT alter the line/key before returning to the Item part.

In fact, if the file structure is so simple and isn't planned to have any evolution or added depth level, you can even do it without recursivity and simply nest three "while" loops, each loop maintaining an index corresponding to the current depth level (Item, SubSector or Data) and creating corresponding arrays once the inner loop exits.

user_142834890
Related