parsing xbrl with python and regular expression to find TextBlocks

user_5369777 0
Viewed 1249

I'm using python and ElementTree to access a list of of .xml files scraped from EDGAR. I've read and re-read the ElementTree/python.org page and am still not understanding how to drill down into the data. How am I supposed to use ElementTree to get something like the first TextBlock for the listed .xmls

import import re
from urllib2 import urlopen
import requests
import xml.etree.ElementTree as ET
full_xml =['https://www.sec.gov/Archives/edgar/data/1593001/000121390017010242/ngtf-20170630.xml', 'https://www.sec.gov/Archives/edgar/data/13573/000143774917016692/bwla-20170702.xml', 'https://www.sec.gov/Archives/edgar/data/1652871/000165287117000030/none-20170630.xml', 'https://www.sec.gov/Archives/edgar/data/1434674/000154972717000042/chnd-20170630_cal.xml', 'https://www.sec.gov/Archives/edgar/data/1083922/000130841117000030/arao-20170331.xml']
for xml in full_xml:
    file = urllib2.urlopen(xml)
    tree = ET.parse(file)
    root = tree.getroot()
    print root
xbrl web-scraping xml python
1 Answers

The information to find textblocks is not only in the XBRL instance (main .xml file). It is also in the taxonomy schema files that belong to the DTS.

Finding textblock facts at the level of XML would require:

  • constructing the DTS by resolving all links to schemas and linkbases from the instance

  • building a list of concepts gathered from all the schemas found, together with their metadata

  • filter these concepts by type (you want to find those with type nonnum:textBlockItemType -- namespace-sensitive comparison)

  • lookup the facts in the XBRL instance that are associated to a concept that made it through the above filter

  • potentially dealing with dimensions to only include dimensionless facts

This would would be theoretically doable, but it would be very complex and resource consuming to do at the level of XML, and prone to errors -- even more so using a library within an imperative language outside of the XML technology stack (such as XQuery). In fact, this amounts to reimplementing a (partial) XBRL processor and this is beyond what regular expressions can do.

In general, I strongly recommend using an existing XBRL processor -- there are open source processors out there, some may even be compatible with python -- where the above logics is already implemented, and it suffices to use an API (e.g., REST or python) to browse through concepts, select text blocks, and lookup the facts with the appropriate data model.

The XBRL technology stack is still at its debuts and many processors are still not dealing with dimensions at the appropriate abstraction level, but if it continues gaining popularity the number of products should increase, and they should become more complete and stable.

user_2842850
Related