I have many rows in XML and I'm trying to get instances of a particular node attribute.
<foo>
<bar>
<type foobar="1"/>
<type foobar="2"/>
</bar>
</foo>
How do I access the values of the attribute foobar? In this example, I want "1" and "2".
I suggest ElementTree. There are other compatible implementations of the same API, such as lxml, and cElementTree in the Python standard library itself; but, in this context, what they chiefly add is even more speed -- the ease of programming part depends on the API, which ElementTree defines.
First build an Element instance root from the XML, e.g. with the XML function, or by parsing a file with something like:
import xml.etree.ElementTree as ET
root = ET.parse('thefile.xml').getroot()
Or any of the many other ways shown at ElementTree. Then do something like:
for type_tag in root.findall('bar/type'):
value = type_tag.get('foobar')
print(value)
Output:
1
2
minidom is the quickest and pretty straight forward.
XML:
<data>
<items>
<item name="item1"></item>
<item name="item2"></item>
<item name="item3"></item>
<item name="item4"></item>
</items>
</data>
Python:
from xml.dom import minidom
dom = minidom.parse('items.xml')
elements = dom.getElementsByTagName('item')
print(f"There are {len(elements)} items:")
for element in elements:
print(element.attributes['name'].value)
Output:
There are 4 items:
item1
item2
item3
item4
You can use BeautifulSoup:
from bs4 import BeautifulSoup
x="""<foo>
<bar>
<type foobar="1"/>
<type foobar="2"/>
</bar>
</foo>"""
y=BeautifulSoup(x)
>>> y.foo.bar.type["foobar"]
u'1'
>>> y.foo.bar.findAll("type")
[<type foobar="1"></type>, <type foobar="2"></type>]
>>> y.foo.bar.findAll("type")[0]["foobar"]
u'1'
>>> y.foo.bar.findAll("type")[1]["foobar"]
u'2'
lxml.objectify is really simple.
Taking your sample text:
from lxml import objectify
from collections import defaultdict
count = defaultdict(int)
root = objectify.fromstring(text)
for item in root.bar.type:
count[item.attrib.get("foobar")] += 1
print dict(count)
Output:
{'1': 1, '2': 1}
Python has an interface to the expat XML parser.
xml.parsers.expat
It's a non-validating parser, so bad XML will not be caught. But if you know your file is correct, then this is pretty good, and you'll probably get the exact info you want and you can discard the rest on the fly.
stringofxml = """<foo>
<bar>
<type arg="value" />
<type arg="value" />
<type arg="value" />
</bar>
<bar>
<type arg="value" />
</bar>
</foo>"""
count = 0
def start(name, attr):
global count
if name == 'type':
count += 1
p = expat.ParserCreate()
p.StartElementHandler = start
p.Parse(stringofxml)
print count # prints 4
XML:
<foo>
<bar>
<type foobar="1"/>
<type foobar="2"/>
</bar>
</foo>
Python code:
import xml.etree.cElementTree as ET
tree = ET.parse("foo.xml")
root = tree.getroot()
root_tag = root.tag
print(root_tag)
for form in root.findall("./bar/type"):
x=(form.attrib)
z=list(x)
for i in z:
print(x[i])
Output:
foo
1
2
There's no need to use a lib specific API if you use python-benedict. Just initialize a new instance from your XML and manage it easily since it is a dict subclass.
Installation is easy: pip install python-benedict
from benedict import benedict as bdict
# data-source can be an url, a filepath or data-string (as in this example)
data_source = """
<foo>
<bar>
<type foobar="1"/>
<type foobar="2"/>
</bar>
</foo>"""
data = bdict.from_xml(data_source)
t_list = data['foo.bar'] # yes, keypath supported
for t in t_list:
print(t['@foobar'])
It supports and normalizes I/O operations with many formats: Base64, CSV, JSON, TOML, XML, YAML and query-string.
It is well tested and open-source on GitHub. Disclosure: I am the author.
These are some pros of the two most used libraries I would have benefit to know before choosing between them.
standalone="no"? .node.sourceline allows to easily get the line of the XML element you are using.simplified_scrapy: a new lib, I fell in love with it after I used it. I recommend it to you.
from simplified_scrapy import SimplifiedDoc
xml = '''
<foo>
<bar>
<type foobar="1"/>
<type foobar="2"/>
</bar>
</foo>
'''
doc = SimplifiedDoc(xml)
types = doc.selects('bar>type')
print (len(types)) # 2
print (types.foobar) # ['1', '2']
print (doc.selects('bar>type>foobar()')) # ['1', '2']
Here are more examples. This lib is easy to use.
#If the xml is in the form of a string as shown below then
from lxml import etree, objectify
'''sample xml as a string with a name space {http://xmlns.abc.com}'''
message =b'<?xml version="1.0" encoding="UTF-8"?>\r\n<pa:Process xmlns:pa="http://xmlns.abc.com">\r\n\t<pa:firsttag>SAMPLE</pa:firsttag></pa:Process>\r\n' # this is a sample xml which is a string
print('************message coversion and parsing starts*************')
message=message.decode('utf-8')
message=message.replace('<?xml version="1.0" encoding="UTF-8"?>\r\n','') #replace is used to remove unwanted strings from the 'message'
message=message.replace('pa:Process>\r\n','pa:Process>')
print (message)
print ('******Parsing starts*************')
parser = etree.XMLParser(remove_blank_text=True) #the name space is removed here
root = etree.fromstring(message, parser) #parsing of xml happens here
print ('******Parsing completed************')
dict={}
for child in root: # parsed xml is iterated using a for loop and values are stored in a dictionary
print(child.tag,child.text)
print('****Derving from xml tree*****')
if child.tag =="{http://xmlns.abc.com}firsttag":
dict["FIRST_TAG"]=child.text
print(dict)
### output
'''************message coversion and parsing starts*************
<pa:Process xmlns:pa="http://xmlns.abc.com">
<pa:firsttag>SAMPLE</pa:firsttag></pa:Process>
******Parsing starts*************
******Parsing completed************
{http://xmlns.abc.com}firsttag SAMPLE
****Derving from xml tree*****
{'FIRST_TAG': 'SAMPLE'}'''
If you don't want to use any external libraries or 3rd party tools, Please try below code.
xml into python dictionary<tag/> and tags with only attributes like <tag var=val/>Code
import re
def getdict(content):
res=re.findall("<(?P<var>\S*)(?P<attr>[^/>]*)(?:(?:>(?P<val>.*?)</(?P=var)>)|(?:/>))",content)
if len(res)>=1:
attreg="(?P<avr>\S+?)(?:(?:=(?P<quote>['\"])(?P<avl>.*?)(?P=quote))|(?:=(?P<avl1>.*?)(?:\s|$))|(?P<avl2>[\s]+)|$)"
if len(res)>1:
return [{i[0]:[{"@attributes":[{j[0]:(j[2] or j[3] or j[4])} for j in re.findall(attreg,i[1].strip())]},{"$values":getdict(i[2])}]} for i in res]
else:
return {res[0]:[{"@attributes":[{j[0]:(j[2] or j[3] or j[4])} for j in re.findall(attreg,res[1].strip())]},{"$values":getdict(res[2])}]}
else:
return content
with open("test.xml","r") as f:
print(getdict(f.read().replace('\n','')))
Sample input
<details class="4b" count=1 boy>
<name type="firstname">John</name>
<age>13</age>
<hobby>Coin collection</hobby>
<hobby>Stamp collection</hobby>
<address>
<country>USA</country>
<state>CA</state>
</address>
</details>
<details empty="True"/>
<details/>
<details class="4a" count=2 girl>
<name type="firstname">Samantha</name>
<age>13</age>
<hobby>Fishing</hobby>
<hobby>Chess</hobby>
<address current="no">
<country>Australia</country>
<state>NSW</state>
</address>
</details>
Output (Beautified)
[
{
"details": [
{
"@attributes": [
{
"class": "4b"
},
{
"count": "1"
},
{
"boy": ""
}
]
},
{
"$values": [
{
"name": [
{
"@attributes": [
{
"type": "firstname"
}
]
},
{
"$values": "John"
}
]
},
{
"age": [
{
"@attributes": []
},
{
"$values": "13"
}
]
},
{
"hobby": [
{
"@attributes": []
},
{
"$values": "Coin collection"
}
]
},
{
"hobby": [
{
"@attributes": []
},
{
"$values": "Stamp collection"
}
]
},
{
"address": [
{
"@attributes": []
},
{
"$values": [
{
"country": [
{
"@attributes": []
},
{
"$values": "USA"
}
]
},
{
"state": [
{
"@attributes": []
},
{
"$values": "CA"
}
]
}
]
}
]
}
]
}
]
},
{
"details": [
{
"@attributes": [
{
"empty": "True"
}
]
},
{
"$values": ""
}
]
},
{
"details": [
{
"@attributes": []
},
{
"$values": ""
}
]
},
{
"details": [
{
"@attributes": [
{
"class": "4a"
},
{
"count": "2"
},
{
"girl": ""
}
]
},
{
"$values": [
{
"name": [
{
"@attributes": [
{
"type": "firstname"
}
]
},
{
"$values": "Samantha"
}
]
},
{
"age": [
{
"@attributes": []
},
{
"$values": "13"
}
]
},
{
"hobby": [
{
"@attributes": []
},
{
"$values": "Fishing"
}
]
},
{
"hobby": [
{
"@attributes": []
},
{
"$values": "Chess"
}
]
},
{
"address": [
{
"@attributes": [
{
"current": "no"
}
]
},
{
"$values": [
{
"country": [
{
"@attributes": []
},
{
"$values": "Australia"
}
]
},
{
"state": [
{
"@attributes": []
},
{
"$values": "NSW"
}
]
}
]
}
]
}
]
}
]
}
]
If the source is an xml file, say like this sample
<pa:Process xmlns:pa="http://sssss">
<pa:firsttag>SAMPLE</pa:firsttag>
</pa:Process>
you may try the following code
from lxml import etree, objectify
metadata = 'C:\\Users\\PROCS.xml' # this is sample xml file the contents are shown above
parser = etree.XMLParser(remove_blank_text=True) # this line removes the name space from the xml in this sample the name space is --> http://sssss
tree = etree.parse(metadata, parser) # this line parses the xml file which is PROCS.xml
root = tree.getroot() # we get the root of xml which is process and iterate using a for loop
for elem in root.getiterator():
if not hasattr(elem.tag, 'find'): continue # (1)
i = elem.tag.find('}')
if i >= 0:
elem.tag = elem.tag[i+1:]
dict={} # a python dictionary is declared
for elem in tree.iter(): #iterating through the xml tree using a for loop
if elem.tag =="firsttag": # if the tag name matches the name that is equated then the text in the tag is stored into the dictionary
dict["FIRST_TAG"]=str(elem.text)
print(dict)
Output would be
{'FIRST_TAG': 'SAMPLE'}