How to handle ParseError while reading Large XML file?

jmparejaz 0
Viewed 23

I am learning some begineer level bigdata and trying to read large xml files to convert them to pandas DataFrame using ElemetTree Iterparse however I got an ParserError about invalid character

import numpy as np
import pandas as pd
import xml.etree.ElementTree as et

def read_xml(filepath,tag,*args):
    atts=[arg for arg in args]
    dict_list = []
    
    for _, elem in et.iterparse(filepath, events=("end",)):

        tempdict={}
        
        if elem.tag == tag:
            for i in range(len(atts)):
                tempdict[atts[i]]=elem.attrib[atts[i]]
        
            dict_list.append(tempdict)
            elem.clear()

    return pd.DataFrame(dict_list)

When I check the some lines near the exception I got this:

enter image description here

and I suppose the problem is about those characters 


&#xA

this answer explain what those characters mean, but still I dont know how to solve it. https://stackoverflow.com/a/15722541/5711393

bigdata xml pandas python
1 Answers

I solved this issue with lxml library

from lxml import etree

def read_xml(filepath,tag,*args):
    #my_parser = etree.XMLParser(recover=True)
    atts=[arg for arg in args]
    dict_list = []
    
    for _, elem in etree.iterparse(filepath, events=("end",),recover=True):

        tempdict={}
        
        if elem.tag == tag:
            for i in range(len(atts)):
                tempdict[atts[i]]=elem.attrib[atts[i]]
        
            dict_list.append(tempdict)
            elem.clear()

    return pd.DataFrame(dict_list)
jmparejaz0
Related