How to properly use xmlfile api from lxml

user_10348381 0
Viewed 362

I have a large (5+ gigs) XML file which I need to parse, do some operation & write a new XML file.

dummy.xml

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.namespace.com" catalog-id="test-catalog">
    <header>
        <name>Product Catalog</name>
    </header>

    <product product-id="1234">
        <available-flag>false</available-flag>
        <name>product1</name>
    </product>
    <product product-id="5678">
        <available-flag>false</available-flag>
        <name>product1</name>
    </product>
    <product product-id="9999">
        <available-flag>false</available-flag>
        <name>product1</name>
    </product>    
</catalog>

As you see the above XML has 3 product tags & I need to filter some product-ids on basis of a pre-defined list of ids.

I am using lxml iterparse to parse the XML iteratively & want to use xmlfile API to create a new XML incrementally to keep the memory footprint low. So, my motive is to filter out the product tags which don't meet the criteria & copy the rest of the XML tags as it is.

from lxml import etree
f = './dummy.xml'

f1 = './test.xml'
context = etree.iterparse(f, events=('start',))
productsToExport = ['1234']


with etree.xmlfile(f1, encoding='utf-8') as xf:
    xf.write_declaration()
    with xf.element('catalog xmlns="http://www.namespace.com" catalog-id="test-catalog"'):
        for event, element in context:
            tagName = etree.QName(element.tag).localname
            if (tagName == 'product'):
                pid = element.get('product-id')
                if (pid in productsToExport):
                    xf.write(element)
            elif (tagName == 'header'):
                xf.write(element) # copy existing header tag as it is

Above code works ok & generates a XML as below

<?xml version='1.0' encoding='utf-8'?>
<catalog xmlns="http://www.namespace.com" catalog-id="test-catalog">
    <header xmlns="http://www.namespace.com">
        <name>Product Catalog</name>
    </header>

    <product xmlns="http://www.namespace.com" product-id="1234">
        <available-flag>false</available-flag>
        <name>product1</name>
    </product>
</catalog xmlns="http://www.namespace.com" catalog-id="test-catalog">

If you observe the above XML it has few issues:

  • Closing <catalog> tag has xmlns & catalog-id present in it
  • All tags like header, product have xmlns attribute present in it

I checked xmlfile api documentation but couldn't find a way to fix above issues.

EDIT:

I managed to resolve the 1st issue by using below

attribs = {'xmlns' : 'http://www.namespace.com', 'catalog-id' : 'test-catalog'}
 with xf.element('catalog', attribs):
     # previous logic

So, now am left with removing the namespace from every element.

lxml python-3.x xml python
1 Answers

Consider simply rebuilding the XML tree with lxml.etree methods instead of the xmlfile API, still in the context of your iterparse:

from lxml import etree

f = './dummy.xml'
f1 = './test.xml'    
productsToExport = ['1234']

# ROOT ELEMENT WITH DEFUALT NAMESPACE
my_nmsp = {None: 'http://www.namespace.com'}

# INITIALIZE ITERATOR
context = etree.iterparse(f, events=('start',))

for event, element in context:    
    tagName = etree.QName(element.tag).localname   

    for prod in productsToExport:
        root = etree.Element('catalog', nsmap=my_nmsp)
        root.text = '\n\t'
        root.attrib['catalog-id'] = "test-catalog"

        # PRODUCT ELEMENT
        if tagName == 'product':
            pid = element.get('product-id')

            if pid == prod:
                root.append(element)

        # HEADER ELEMENT        
        elif (tagName == 'header'):
            root.append(element)

        # OUTPUT TREE TO FILE
        with open(f1, 'wb') as f:
            f.write(etree.tostring(root, pretty_print=True))

Output

<catalog xmlns="http://www.namespace.com" catalog-id="test-catalog">
    <header>
        <name>Product Catalog</name>
    </header>

    <product product-id="1234">
        <available-flag>false</available-flag>
        <name>product1</name>
    </product> 
</catalog>
user_14224510
Related