Nokogiri SAX ignoring empty characters

user_4229641 0
Viewed 124

I'm trying to parse a large xml file using sax parser. When the parser reaches a node which is empty the characters method doesn't fire. Here is an example...

require 'nokogiri'

class Parser < Nokogiri::XML::SAX::Document
  def initialize
    @count=1
  end
  def start_element(name, attrs = [])
    puts name
  end
  def characters(string)
    string.strip!
    puts "#{@count} #{string}"
    @count += 1
  end
  def end_element(name)
    puts name
  end
end

Nokogiri::XML::SAX::Parser.new(Parser.new).parse(File.open('sax_example3.xml'))

Here is the sample xml document.

<?xml version="1.0" encoding="UTF-8"?>
<root>
    <ISA type="array">
        <ISA>
            <I02>
                <name>Information1</name>
                <value>
                    <raw>00</raw>
                    <description></description>
                </value>
            </I02>
            <I02>
                <name>Information2</name>
                <value>
                    <raw></raw>
                    <description nil="true"/>
                </value>
            </I02>
        </ISA>
    </ISA>
</root>

I had to use sax because the file has around 6.5 million lines.

What I'm trying to do is gather all the name values and then raw values into separate arrays and later I can zip both arrays to get key value pairs.

Am I approaching this the correct way? Is there any other way to do this?

Edit:

What I expected

array1 = ["Information1","Information2"]  
array2 = ["00", ""]

All name values are assigned to array1 and raw values to array2 like shown above.

What I'm getting

array1 = ["Information1","Information2"]  
array2 = ["00"]

array2 doesn't have the same number of elements as array1, which means there is no way to map names to array. The reason for this I think is that the characters method is not called if the node is empty.

Here is the output of the above program (Edited the above script and added line numbers)

root           
1              
ISA            
2              
ISA            
3              
I02            
4              
name           
5 Information1 
name           
6              
value          
7              
raw            
8 00           
raw            
9              
description    
description    
10             
value          
11             
I02            
12             
I02            
13             
name           
14 Information2
name           
15             
value          
16             
raw            
raw            
17             
description    
description    
18             
value          
19             
I02            
20             
ISA            
21             
ISA            
22             
root

As you can see, between lines (9 & 10), (16 & 17) and (17 & 18) the start_element & end_element methods are executed but the characters method isn't.

ruby xml
1 Answers

Since characters might not get called, you'll need to instead watch for the <name> and <raw> elements themselves. If we can assume <name> and <raw> always come in pairs and in that order, we can create a new "empty" pair (e.g. { name: nil, raw: nil }) every time we run into the former, and then fill in the values when (and if) characters is called:

class Parser < Nokogiri::XML::SAX::Document
  def initialize(*args)
    @vals = []
    @current_el = nil
    super
  end

  def start_element(el_name, attrs = [])
    if el_name == "name"
      @vals << { name: nil, raw: nil }
      @current_el = "name"
    elsif el_name == "raw"
      @current_el = "raw"
    else
      @current_el = nil
    end
  end

  def end_element(el_name)
    if el_name == "name" || el_name == "raw"
      @current_el = nil
    end
  end

  def characters(str)
    str = str.strip
    if @current_el == "name"
      @vals.last[:name] = str
    elsif @current_el == "raw"
      @vals.last[:raw] = str
    end
  end

  def end_document
    pp @vals
  end
end

You can see this in action on repl.it (but be forewarned, it takes forever to run the first time, because Nokogiri): https://repl.it/@jrunning/SpitefulRichLists

user_1791250
Related