XML processing in distributed system

user_12880432 0
Viewed 105

I'm working on Big data from last couple of months.

1

I'm curious to know how XML processing works in multi nodes network. Lets say I have one xml and that's having repetitive tags and in every those tags there are more similar tags.

To process XML, Whole XML needs to be in memory first. So, the moment I submit my hadoop mapreduce task to process 100 GB of XML in the cluster (Lets say 5 node cluster) to process, then how loading of XML in different nodes happens ?

In case of flat files, I can imagine that keep reading the records( one by one) from HDFS and distributing among nodes. But not able to understand in XML. How hadoops know, on what basis of tags splits or distribution needs to happen ?

<employees>
    <employee id="be129">
        <firstname>Jane</firstname>
        <lastname>Doe</lastname>
        <title>Engineer</title>
        <division>Materials</division>
        <building>327</building>
        <room>19</room>
        <supervisor>be131</supervisor>
    </employee>
    <employee id="be130">
        <firstname>William</firstname>
        <lastname>Defoe</lastname>
        <title>Accountant</title>
        <division>Accts Payable</division>
        <building>326</building>
        <room>14a</room>
    </employee>
    <employee id="be131">
        <firstname>Jack</firstname>
        <lastname>Dee</lastname>
        <title>Engineering Manager</title>
        <division>Materials</division>
        <building>327</building>
        <room>21</room>
    </employee>
    <employee id="be132">
        <firstname>Sandra</firstname>
        <lastname>Rogers</lastname>
        <title>Engineering</title>
        <division>Materials</division>
        <building>327</building>
        <room>22</room>
    </employee>
    <employee id="be133">
        <firstname>Steve</firstname>
        <lastname>Casey</lastname>
        <title>Engineering</title>
        <division>Materials</division>
        <building>327</building>
        <room>24</room>
    </employee>
    <employee id="be135">
        <firstname>Michelle</firstname>
        <lastname>Michaels</lastname>
        <title>COO</title>
        <division>Management</division>
        <building>216</building>
        <room>264</room>
    </employee>
</employees>

Lets consider above XML as an example, and imagine there are 10 million of employee tags, Then how hadoop mapper splitting mechanism works to distribute xml among 5 nodes and process and coordinate upon distributed XML to process ?

I hope my question is clear to all.

hadoop xml
2 Answers

So, first of all, don't use MapReduce. At least, not default options because it only reads line delimited files, not XML formatting

If you want to process xml in parallel, opt for something like spark-xml or pre-process the XML documents before they even enter HDFS into a more Hadoop friendly format, for example, using something like Apache Nifi XML/XPath processors

user_23086830

The XML parsing pretty well has to be a sequential process, but it doesn't have to build the whole document in memory.

Everything that happens after parsing can be distributed and parallel.

You could do the initial processing of the XML using a SAX parser or using XSLT 3.0 streaming to split it up into smaller chunks, and then send the smaller chunks to multiple parallel processes. I don't know if any frameworks like NiFi will do this for you, or if you have to code it yourself "by hand".

user_4154480
Related