Python Beautiful Soup XML Conditional Query (mixed tags and attributes) Parsing

user_2204984 0
Viewed 42

I have two versions of XML files that I need to extract content from. Both have the same information in two different formats as follows (not just different tags, but different structure):

  1. The first one has active and inactive elements defined between: <activeelementSubstance></activeelementSubstance> and <inactiveelementSubstance></inactiveelementSubstance>
  2. The second has active and inactive elements defined between: <element classCode="IACT"></element classCode="IACT"> and <element classCode="ACTIM"></element classCode="ACTIM">

How do I address both situations to extract Inactive and Active Elements ? (there are several examples where tags have synonyms, but I have not seen any where the actual structure is different as in this case)

I came up with the following code (not very clean) to extract from the second case (extracting the element and its code):

activeElements = soup.findAll('Element', attrs={'classCode': 'ACTIM'})
for i in activeElements:
    aiName = i.find('name')
    aiCode = str(i.find('code'))
    print(aiName.text)
    print( re.findall(r'"(.*?)"', aiCode)[0] )


print('\nInactive Elements\n')

inactiveElements = soup.findAll('Element', attrs={'classCode': 'IACT'})
for i in inactiveElements:
    aiName = i.find('name')
    print(aiName.text)
    aiCode = i.find('code')['code']
    print(aiCode)

Examples of XML files are as follows:

First type (with the format <element classCode="IACT"></element classCode="IACT"> and <element classCode="ACTIM"></element classCode="ACTIM">):

<?xml version="1.0" encoding="UTF-8"?>
<document>
<manufacturedProduct>
<element classCode="IACT">
<elementSubstance>
<code code="36SFW2JZ" codeSystem="33590coding"/>
<name>HYPROMELLOSE 2910 (15 MPA.S)</name>
</elementSubstance>
</element>
<element classCode="IACT">
<elementSubstance>
<code code="70097M6I" codeSystem="33590coding"/>
<name>MAGNESIUM STEARATE</name>
</elementSubstance>
</element>
<elementSubstance>
<code code="XHX3C3X6" codeSystem="33590coding"/>
<name>TRIACETIN</name>
</elementSubstance>
</element>
<element classCode="ACTIM">
<quantity>
<numerator unit="mg" value="250"/>
<denominator unit="1" value="1"/>
</quantity>
<elementSubstance>
<code code="JTE4MNN1" codeSystem="33590coding"/>
<name>AZITHROMYCIN MONOHYDRATE</name>
</elementSubstance>
</element>
</manufacturedProduct>
</document>

Second type (with the format<activeelementSubstance></activeelementSubstance> and <inactiveelementSubstance></inactiveelementSubstance>):

<?xml version="1.0" encoding="UTF-8"?>
<document>
<manufacturedProduct>
<activeelementSubstance>
<code code="VB0R961H" codeSystem="33590coding" codeSystemName="USDA" />
<name>Prednisone</name>
</activeelementSubstance>
</activeelement>
<inactiveelement>
<inactiveelementSubstance>
<code code="776XM704" codeSystem="33590coding" codeSystemName="USDA" />
<name>calcium stearate</name>
</inactiveelementSubstance>
</inactiveelement>
<inactiveelement>
<inactiveelementSubstance>
<name>corn starch</name>
</inactiveelementSubstance>
</inactiveelement>
</manufacturedProduct>
</document>

The XML files are deeply nested (This is in part why I am using Beautiful soup), I tried cleaning and extracting the relevant portion of them.

beautifulsoup xml parsing python
2 Answers

You could use CSS OR syntax within a CSS selector list than add conditional logic to determine where to assign extracted results. The below could do with a refactor to reduce nesting e.g. move certain parts into their own functions, but it gives a starting point.

from bs4 import BeautifulSoup as bs

type_1 = '''
<?xml version="1.0" encoding="UTF-8"?>
<document>
<manufacturedProduct>
<element classCode="IACT">
<elementSubstance>
<code code="36SFW2JZ" codeSystem="33590coding"/>
<name>HYPROMELLOSE 2910 (15 MPA.S)</name>
</elementSubstance>
</element>
<element classCode="IACT">
<elementSubstance>
<code code="70097M6I" codeSystem="33590coding"/>
<name>MAGNESIUM STEARATE</name>
</elementSubstance>
</element>
<elementSubstance>
<code code="XHX3C3X6" codeSystem="33590coding"/>
<name>TRIACETIN</name>
</elementSubstance>
</element>
<element classCode="ACTIM">
<quantity>
<numerator unit="mg" value="250"/>
<denominator unit="1" value="1"/>
</quantity>
<elementSubstance>
<code code="JTE4MNN1" codeSystem="33590coding"/>
<name>AZITHROMYCIN MONOHYDRATE</name>
</elementSubstance>
</element>
</manufacturedProduct>
</document>'''

type_2 = '''
<?xml version="1.0" encoding="UTF-8"?>
<document>
<manufacturedProduct>
<activeelementSubstance>
<code code="VB0R961H" codeSystem="33590coding" codeSystemName="USDA" />
<name>Prednisone</name>
</activeelementSubstance>
</activeelement>
<inactiveelement>
<inactiveelementSubstance>
<code code="776XM704" codeSystem="33590coding" codeSystemName="USDA" />
<name>calcium stearate</name>
</inactiveelementSubstance>
</inactiveelement>
<inactiveelement>
<inactiveelementSubstance>
<name>corn starch</name>
</inactiveelementSubstance>
</inactiveelement>
</manufacturedProduct>
</document>'''


results = {'active': [], 'inactive': []}

for doc in [type_1, type_2]:
    
    soup = bs(doc, 'lxml')
    
    for i in soup.select('[classcode=IACT], [classcode=ACTIM], activeelement, inactiveelement'):

        if i.get('classcode') == 'IACT' or i.name == 'inactiveelement':
            try:
                d = {i.select_one('name').text:i.select_one('code')['code']}
            except:
                d = {i.select_one('name').text:'missing'}
            results['inactive'].append(d)
        else:
            results['active'].append({i.select_one('name').text:i.select_one('code')['code']})
            
print(results)
user_62412350

You can use CSS selectors with ,. For example:

from bs4 import BeautifulSoup

xml_doc1 = ...version 1 of the xml document...
xml_doc2 = ...version 2 of the xml document...


soup = BeautifulSoup(xml_doc1 + xml_doc2, "html.parser")

elem = soup.select(
    'element[classCode="IACT"], element[classCode="ACTIM"], activeelementSubstance, inactiveelementSubstance'
)

for e in elem:
    code = c["code"] if (c := e.code) else "-"
    name = n.text if (n := e.find("name")) else "-"

    print(code, name)

Prints:

36SFW2JZ HYPROMELLOSE 2910 (15 MPA.S)
70097M6I MAGNESIUM STEARATE
JTE4MNN1 AZITHROMYCIN MONOHYDRATE
VB0R961H Prednisone
776XM704 calcium stearate
- corn starch

Or split between active and inactive:

print("Active Elements")

elem = soup.select('element[classCode="ACTIM"], activeelementSubstance')

for e in elem:
    code = c["code"] if (c := e.code) else "-"
    name = n.text if (n := e.find("name")) else "-"

    print(code, name)


print("\nInactive Elements")

elem = soup.select('element[classCode="IACT"], inactiveelementSubstance')

for e in elem:
    code = c["code"] if (c := e.code) else "-"
    name = n.text if (n := e.find("name")) else "-"

    print(code, name)
user_100359850
Related