How to parse big XML in google cloud function efficiently?

user_3253917 0
Viewed 451

I have to extract data from XML files with the size of several hundreds of MB in a Google Cloud Function and I was wondering if there are any best practices?

Since I am used to nodejs I was looking at some popular libraries like fast-xml-parser but it seems cumbersome if you only want specific data from a huge xml. I am also not sure if there are any performance issues when the XML is too big. Overall this does not feel like the best solution to parse and extract data from huge XMLs.

Then I was wondering if I could use BigQuery for this task where I simple convert the xml to json and throw it into a Dataset where I then can use a query to retrieve the data I want.

Another solution could be to use python for the job since it is good in parsing and extracting data from a XML so even though I have no experience in python I was wondering if this path could still be the best solution?

If anything above does not make sense or if one solution is preferable to the other or if anyone can share any insights I would highly appreciate it!

google-cloud-functions google-bigquery node.js xml-parsing
1 Answers

I suggest you to check this article in which they discuss how to load XML data into BigQuery using Python Dataflow. I think that this approach may work in your situation.

Basically what they suggest is:

  • To parse the xml into a Python dictionary using the package xmltodict.
  • Specify a schema of the output table in BigQuery.
  • Use a Beam pipeline to take an XML file and use it to populate a BigQuery table.
user_93302080
Related