I'm using the Nokogiri gem in Ruby and running into some problems.
I want to scrape addresses from webpages and there is no set format to the way the addresses will be displayed.
I've got a list of postcodes and I want my Ruby script to return the node including the postcode so that I can find the rest of the address.
This is what I've got in Ruby, with some example HTML content:
require 'nokogiri'
require 'open-uri'
content1 = '
<div>
<div>
<div>Our Address:</div>
1 North Street
North Town
North County
N21 4DD
</div>
</div>'
doc = Nokogiri::HTML(content1)
result = doc.search "[text()*='N21 4DD']"
puts result.inspect
This returns []
I understand the example above is a strange way for an address to appear in HTML but it's the simplest way I can show the problems I've had. Here's another content variable that returns nothing:
content1 = '
<div>
<div>Our Address:</div>
<div>
1 North Street<br>
North Town<br>
North County<br>
N21 4DD
</div>
</div>'
I know that Nokogiri might have trouble with the above because the <br> tags should be </br> but this is quite common on websites.
THIS EXAMPLE WORKS:
content1 = '
<div>
<div>Our Address:</div>
<div>
1 North Street
North Town
North County
N21 4DD
</div>
</div>'
Can someone explain why the node is not being found from the first two content examples above and how I can fix this?
I'm not looking for a custom solution that will find the postcode in the sample content examples above – these are just for demonstration purposes. The postcode (and address) could be anywhere in the html – body, p, div, td, span, li etc.
Thanks.