Kimurai scraping inconsistent returning results

Question

I'm attempting to use Kimurai to get product information off of a local grocery website. The problem I have is it will only actually grab the data maybe 1 out of 10 times. I have already built the scraper for the individual product pages and found that unless I used sleep for 5 seconds that the results were inconsistent. I've been playing around with this and even setting it up to 30 seconds (much longer than the page takes to load for me) it still doesn't work consistently and anything over 30 seconds seems to time out the search.

My goal is to have Kimurai load the search the page (https://www.loblaws.ca/search?search-bar=[searchterm]) and gather the Name, and price of the products (and eventually the URL of the product page, but I haven't got that far yet)

Controller Code:

def search
     searchterm = params[:lookup]
     url = 'https://www.loblaws.ca/search?search-bar=' + searchterm     
    response = FoodsearchSpider.process(url)
    if response[:status] == :completed && response[:error].nil?
      flash.now[:notice] = "Successfully scraped url"
    else
      flash.now[:alert] = response[:error]
    end
      rescue StandardError => e
    flash.now[:alert] = "Error: #{e}"
    end

Model Code: https://pastebin.com/Aq2G9BFL

class FoodsearchSpider < Kimurai::Base 
 @name = 'foodsearch_spider'
  @engine = :selenium_chrome
  @config = {
      user_agent: "Mozilla/5.0 Firefox/61.0",
  }
 
  def self.process(url)
    @start_urls = [url]
    self.crawl!
  end
 
  def parse(response, url:, data: {})
      sleep 30
      response.xpath("//li[@class='product-tile-group__list__item']").each do |result|
        puts "in the loop"
        item = {}
        item[:name]      = result.xpath("//span[@class='product-name product-name--product-tile']")&.text&.squish
        item[:price]    = result.xpath("//*[@id='site-content']/div/div/div/div[2]/div/div[2]/div[2]/div[2]/div/ul/li[1]/div/div/div[3]/div[3]/ul[1]/li[1]/span/span[1]")&.text&.squish
      puts "**********"
      puts item
      puts "***********"
      end
  end
end

everything consistently works fine up until

response.xpath("//li[@class='product-tile-group__list__item']").each do |result|

If it goes into the loop it will work fine but that only happens occasionally.

When it doesn't go into the loop it still reports that it completed successfully, but it never enters the loop or sets the array with the values.

Console log from failure:

Processing by IngredientsController#search as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"7fcXSWF4IAajptTR5cCfLu+cswbWniSbiBvAcCQvVvwaa66Ce2dOfmevx38c7ApXDMBHGaJFQEyhJUg5sjLitg==", "lookup"=>"peppers", "commit"=>"Submit"}
  User Load (0.2ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = ? ORDER BY "users"."id" ASC LIMIT ?  [["id", 1], ["LIMIT", 1]]
searching URL
****************
https://www.loblaws.ca/search?search-bar=peppers
****************
I, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Spider: started: foodsearch_spider
D, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340] DEBUG -- foodsearch_spider: BrowserBuilder (selenium_chrome): created browser instance
D, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340] DEBUG -- foodsearch_spider: BrowserBuilder (selenium_chrome): enabled custom user_agent
D, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340] DEBUG -- foodsearch_spider: BrowserBuilder (selenium_chrome): enabled native headless_mode
I, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Browser: started get request to: https://www.loblaws.ca/search?search-bar=peppers
2021-03-20 19:40:18 WARN Selenium [DEPRECATION] :driver_path is deprecated. Use :service with an instance of Selenium::WebDriver::Service instead.
I, [2021-03-20 19:40:20 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Browser: finished get request to: https://www.loblaws.ca/search?search-bar=peppers
I, [2021-03-20 19:40:20 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Info: visits: requests: 1, responses: 1
D, [2021-03-20 19:40:22 +0000#27612] [C: 47268811491340] DEBUG -- foodsearch_spider: Browser: driver.current_memory: 255830
I, [2021-03-20 19:40:52 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Browser: driver selenium_chrome has been destroyed
I, [2021-03-20 19:40:52 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Spider: stopped: {:spider_name=>"foodsearch_spider", :status=>:completed, :error=>nil, :environment=>"development", :start_time=>2021-03-20 19:40:18 +0000, :stop_time=>2021-03-20 19:40:52 +0000, :running_time=>"34s", :visits=>{:requests=>1, :responses=>1}, :items=>{:sent=>0, :processed=>0}, :events=>{:requests_errors=>{}, :drop_items_errors=>{}, :custom=>{}}}
  Rendering ingredients/search.html.erb within layouts/application
  Rendered ingredients/search.html.erb within layouts/application (0.9ms)
Completed 200 OK in 34363ms (Views: 63.6ms | ActiveRecord: 0.2ms)

Occasionally this actually works and it starts running the code inside the loop (the xpaths inside the code aren't set correctly yet) but it's like maybe 10 percent of the time.

Kimurai scraping inconsistent returning results

0 Answers