Parsing Huge XML Files in PHP

There are only two php APIs that are really suited for processing large files. The first is the old expat api, and the second is the newer XMLreader functions. These apis read continuous streams rather than loading the entire tree into memory (which is what simplexml and DOM does).

For an example, you might want to look at this partial parser of the DMOZ-catalog:

<?php

class SimpleDMOZParser
{
    protected $_stack = array();
    protected $_file = "";
    protected $_parser = null;

    protected $_currentId = "";
    protected $_current = "";

    public function __construct($file)
    {
        $this->_file = $file;

        $this->_parser = xml_parser_create("UTF-8");
        xml_set_object($this->_parser, $this);
        xml_set_element_handler($this->_parser, "startTag", "endTag");
    }

    public function startTag($parser, $name, $attribs)
    {
        array_push($this->_stack, $this->_current);

        if ($name == "TOPIC" && count($attribs)) {
            $this->_currentId = $attribs["R:ID"];
        }

        if ($name == "LINK" && strpos($this->_currentId, "Top/Home/Consumer_Information/Electronics/") === 0) {
            echo $attribs["R:RESOURCE"] . "\n";
        }

        $this->_current = $name;
    }

    public function endTag($parser, $name)
    {
        $this->_current = array_pop($this->_stack);
    }

    public function parse()
    {
        $fh = fopen($this->_file, "r");
        if (!$fh) {
            die("Epic fail!\n");
        }

        while (!feof($fh)) {
            $data = fread($fh, 4096);
            xml_parse($this->_parser, $data, feof($fh));
        }
    }
}

$parser = new SimpleDMOZParser("content.rdf.u8");
$parser->parse();

This is an old post, but first in the google search result, so I thought I post another solution based on this post:

http://drib.tech/programming/parse-large-xml-files-php

This solution uses both XMLReader and SimpleXMLElement :

$xmlFile = 'the_LARGE_xml_file_to_load.xml'
$primEL  = 'the_name_of_your_element';

$xml     = new XMLReader();
$xml->open($xmlFile);

// finding first primary element to work with
while($xml->read() && $xml->name != $primEL){;}

// looping through elements
while($xml->name == $primEL) {
    // loading element data into simpleXML object
    $element = new SimpleXMLElement($xml->readOuterXML());

    // DO STUFF

    // moving pointer   
    $xml->next($primEL);
    // clearing current element
    unset($element);
} // end while

$xml->close();

I would suggest using a SAX based parser rather than DOM based parsing.

Info on using SAX in PHP: http://www.brainbell.com/tutorials/php/Parsing_XML_With_SAX.htm

This isn't a great solution, but just to throw another option out there:

You can break many large XML files up into chunks, especially those that are really just lists of similar elements (as I suspect the file you're working with would be).

e.g., if your doc looks like:

<dmoz>
  <listing>....</listing>
  <listing>....</listing>
  <listing>....</listing>
  <listing>....</listing>
  <listing>....</listing>
  <listing>....</listing>
  ...
</dmoz>

You can read it in a meg or two at a time, artificially wrap the few complete <listing> tags you loaded in a root level tag, and then load them via simplexml/domxml (I used domxml, when taking this approach).

Frankly, I prefer this approach if you're using PHP < 5.1.2. With 5.1.2 and higher, XMLReader is available, which is probably the best option, but before that, you're stuck with either the above chunking strategy, or the old SAX/expat lib. And I don't know about the rest of you, but I HATE writing/maintaining SAX/expat parsers.

Note, however, that this approach is NOT really practical when your document doesn't consist of many identical bottom-level elements (e.g., it works great for any sort of list of files, or URLs, etc., but wouldn't make sense for parsing a large HTML document)

You can combine XMLReader with DOM for this. In PHP both APIs (and SimpleXML) are based on the same library - libxml2. Large XMLs are a list of records typically. So you use XMLReader to iterate the records, load a single record into DOM and use DOM methods and Xpath to extract values. The key is the method XMLReader::expand(). It loads the current node in an XMLReader instance and its descendants as DOM nodes.

Example XML:

<books>
  <book>
    <title isbn="978-0596100087">XSLT 1.0 Pocket Reference</title>
  </book>
  <book>
    <title isbn="978-0596100506">XML Pocket Reference</title>
  </book>
  <!-- ... -->
</books>

Example code:

// open the XML file
$reader = new XMLReader();
$reader->open('books.xml');

// prepare a DOM document
$document = new DOMDocument();
$xpath = new DOMXpath($document);

// find the first `book` element node at any depth
while ($reader->read() && $reader->localName !== 'book') {
  continue;
}

// as long as here is a node with the name "book"
while ($reader->localName === 'book') {
  // expand the node into the prepared DOM
  $book = $reader->expand($document);
  // use Xpath expressions to fetch values
  var_dump(
    $xpath->evaluate('string(title/@isbn)', $book),
    $xpath->evaluate('string(title)', $book)
  );
  // move to the next book sibling node
  $reader->next('book');
}
$reader->close();

Take note that the expanded node is never appended to the DOM document. It allows the GC to clean it up.

This approach works with XML namespaces as well.

$namespaceURI = 'urn:example-books';

$reader = new XMLReader();
$reader->open('books.xml');

$document = new DOMDocument();
$xpath = new DOMXpath($document);
// register a prefix for the Xpath expressions
$xpath->registerNamespace('b', $namespaceURI);

// compare local node name and namespace URI
while (
  $reader->read() &&
  (
    $reader->localName !== 'book' ||
    $reader->namespaceURI !== $namespaceURI
  )
) {
  continue;
}

// iterate the book elements 
while ($reader->localName === 'book') {
  // validate that they are in the namespace
  if ($reader->namespaceURI === $namespaceURI) {
    $book = $reader->expand($document);
    var_dump(
      $xpath->evaluate('string(b:title/@isbn)', $book),
      $xpath->evaluate('string(b:title)', $book)
    );
  }
  $reader->next('book');
}
$reader->close();

I've written a wrapper for XMLReader to (IMHO) make it easier to just get the bits your after. The wrapper allows you to associate a set of paths of the data elements and a callback to be run when this path is found. The path allows regex expressions and also capture groups which can also be passed to the callback.

The library is at https://github.com/NigelRel3/XMLReaderReg and can also be installed using composer require nigelrel3/xml-reader-reg.

An example of how to use it...

$inputFile = __DIR__ ."/../tests/data/simpleTest1.xml";
$reader = new XMLReaderReg\XMLReaderReg();
$reader->open($inputFile);

$reader->process([
    '(.*/person(?:\[\d*\])?)' => function (SimpleXMLElement $data, $path): void {
        echo "1) Value for ".$path[1]." is ".PHP_EOL.
            $data->asXML().PHP_EOL;
    },
    '(.*/person3(\[\d*\])?)' => function (DOMElement $data, $path): void {
        echo "2) Value for ".$path[1]." is ".PHP_EOL.
            $data->ownerDocument->saveXML($data).PHP_EOL;
    },
    '/root/person2/firstname' => function (string $data): void {
        echo "3) Value for /root/person2/firstname is ". $data.PHP_EOL;
    }
    ]);

$reader->close();

As can be seen from the example, you can get the data to be passed as a SimpleXMLElement, a DOMElement or the last one is a string. This will represent only the data which matches the path.

The paths also show how capture groups can be used - (.*/person(?:\[\d*\])?) looks for any person element (including arrays of elements) and $path[1] in the callback displays the path where this particular instance is found.

There is an expanded example in the library as well as unit tests.

I tested the following code with 2 GB xml:

<?php
set_time_limit(0);
$reader = new XMLReader();
if (!$reader->open("data.xml"))
{
    die("Failed to open 'data.xml'");
}
while($reader->read())
{
    $node = $reader->expand();
    // process $node...
}
$reader->close();
?>

My solution:

$reader = new XMLReader();
$reader->open($fileTMP);
 while ($reader->read()) {
 if ($reader->nodeType === XMLReader::ELEMENT && $reader->name === 'xmltag' && $reader->isEmptyElement === false) {
 $item = simplexml_load_string($reader->readOuterXML(), null, LIBXML_NOCDATA); 
   //operations on file
}
}
$reader->close();

Very high performed way is

preg_split('/(<|>)/m', $xmlString);

And after that, only one cycle is needed.