Trying to traverse through nested xml tags but recursive function does not traverse in full depth

Question

I have following xml data in string format, which using lxml package of python, I am parsing into xml.

Now, I have to traverse through this xml data and generate an output in a specific format which will be something like this

And I would like to have the output in the following format:

{
    'B1': 'B_1',
    'B2': 'B_2',
    'B3_B31': 'B3_1',
    'B3_B32': 'B3_2',
    'B3_B33_B331_B3311': '-',
    'B3_B33_B331_B3312': '-',
    'B3_B34_B341_B3411': '-',
    'B3_B34_B342_B3421': '-',
    'B3_B35_B351': 'B35_1',
    'B3_B35_B352_B3521_B35211': '-',
    'B3_B35_B352_B3521_B35212': '-',
    'B3_B36_B361': 'b36_1',
    'B3_B36_B361': 'B36_2',
}

Now, this is just an EXAMPLE. In real scenario, depth of each xml tag might be different. So, I decided to use the recursive approach. And this is so far I have came in terms of code:

class ParseXML:
    main_output = []
    output = {}

    def __init__(self, xml_input):
        parser = ET.XMLParser(recover=True)
        tree = ET.ElementTree(ET.fromstring(xml_input, parser=parser))
        self.root = tree.getroot()

    def parse_outer_xml(self):
        for children in self.root:
            output = self.parse_xml(children, output={})
            self.main_output.append(output)
        return self.main_output

    def parse_xml(self, children, tag=None, output={}):
        if len(children):
            for child in children.getchildren():
                if child.tag.split('}')[1] in GLOBAL_DICT:
                    output['{0}_{1}'.format(tag, child.tag.split('}')[1]) if tag else child.tag.split('}')[1]] = child.text
                else:
                    if child.tag.split('}')[1] not in GLOBAL_EXCLUDE_DICT:
                        if len(child):
                            if children.tag.split('}')[1] == 'B':
                                tag = child.tag.split('}')[1]
                            else:
                                tag = "{0}_{1}".format(tag, child.tag.split('}')[1]) if tag else "{0}_{1}".format(children.tag.split('}')[1], child.tag.split('}')[1])
                            return self.parse_xml(child, tag, output)
                        else:
                            output['{0}_{1}'.format(tag, child.tag.split('}')[1]) if tag else child.tag.split('}')[1]] = child.text if child.text else "-"
        else:
            output['{0}_{1}'.format(tag, children.tag.split('}')[1]) if tag else children.tag.split('}')[1]] = children.text if children.text else "-"
        return output


if __name__ == '__main__':
    parse = ParseXML(data)
    temp = parse.parse_outer_xml()
    pprint(temp)

I am getting this output when I run

[{'B1': 'B_1',
  'B2': 'B_2',
  'B3_B31': 'B3_1',
  'B3_B32': 'B3_2',
  'B3_B33_B331_B3311': '-'}]

But this code does not traverse to full depth. Can anyone look into this and provide some guidance on how to traverse through this xml data till full depth.

user_7326738 · Answer

You can use a recursive generator function:

import xml.etree.ElementTree as ET, re
t = ET.fromstring(re.sub('\sxmlns\="\w+"', '', s_xml))
def flatten(t, p = []):
   if not (c:=list(t)):
      yield ('_'.join(p+[t.tag]), '-' if t.text is None else t.text)
   else:
      yield from [j for k in c for j in flatten(k, p+[t.tag])]

r = dict(j for k in list(t)[0] for j in flatten(k))

Output:

{'B1': 'B_1', 'B2': 'B_2', 'B3_B31': 'B3_1', 'B3_B32': 'B3_2', 'B3_B33_B331_B3311': '-', 'B3_B33_B332_B3321': '-', 'B3_B34_B341_B3411': '-', 'B3_B34_B342_B3421': '-', 'B3_B35_B351': 'B35_1', 'B3_B35_B352_B3521_B35211': '-', 'B3_B36_B361': 'B36_1', 'B3_B36_B362': 'B36_2'}

user_9448090 · Answer

First, the sample xml in your question is not well formed. Assuming that's fixed, you first have to deal with the fact that your xml contains a namespace declaration. So all in all, something like the below (using lxml) should get you at least close enough:

from lxml import etree
doc = etree.XML([your xml above, well formed])

#remove the namespace
for elem in doc.getiterator():
    elem.tag = etree.QName(elem).localname

#from here, get the path of each element and massage it a bit to fit what I believe 
#are your requirements 
tree = etree.ElementTree(doc)    
targets = []
for e in doc.iter():
        path = tree.getpath(e).replace("/A/B/","").replace("/","_") 
        if "A" not in path:        
            if e.text is not None and len(e.text.strip())>0:
                targets.append(path+" : "+e.text.strip())
            else:
                if not e.text:
                    targets.append(path+": -")
                
for target in targets:
    print(target)

Output (at least from the sample xml) should be your expected output.

Trying to traverse through nested xml tags but recursive function does not traverse in full depth

2 Answers