Ruby Nokogiri text search not working with br tags and others

Viewed 365

I'm using the Nokogiri gem in Ruby and running into some problems.

I want to scrape addresses from webpages and there is no set format to the way the addresses will be displayed.

I've got a list of postcodes and I want my Ruby script to return the node including the postcode so that I can find the rest of the address.

This is what I've got in Ruby, with some example HTML content:

require 'nokogiri'
require 'open-uri'

content1 = '
<div>
    <div>
        <div>Our Address:</div>
        1 North Street
        North Town
        North County
        N21 4DD
    </div>
</div>'

doc = Nokogiri::HTML(content1)
result = doc.search "[text()*='N21 4DD']"
puts result.inspect

This returns []

I understand the example above is a strange way for an address to appear in HTML but it's the simplest way I can show the problems I've had. Here's another content variable that returns nothing:

content1 = '
<div>
    <div>Our Address:</div>
    <div>
        1 North Street<br>
        North Town<br>
        North County<br>
        N21 4DD
    </div>
</div>'

I know that Nokogiri might have trouble with the above because the <br> tags should be </br> but this is quite common on websites.

THIS EXAMPLE WORKS:

content1 = '
<div>
    <div>Our Address:</div>
    <div>
        1 North Street
        North Town
        North County
        N21 4DD
    </div>
</div>'

Can someone explain why the node is not being found from the first two content examples above and how I can fix this?

I'm not looking for a custom solution that will find the postcode in the sample content examples above – these are just for demonstration purposes. The postcode (and address) could be anywhere in the html – body, p, div, td, span, li etc.

Thanks.

2 Answers
Related