Scrapy Crawl URLs in Order

user_816224 0
Viewed 37053

So, my problem is relatively simple. I have one spider crawling multiple sites, and I need it to return the data in the order I write it in my code. It's posted below.

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from mlbodds.items import MlboddsItem

class MLBoddsSpider(BaseSpider):
   name = "sbrforum.com"
   allowed_domains = ["sbrforum.com"]
   start_urls = [
       "http://www.sbrforum.com/mlb-baseball/odds-scores/20110328/",
       "http://www.sbrforum.com/mlb-baseball/odds-scores/20110329/",
       "http://www.sbrforum.com/mlb-baseball/odds-scores/20110330/"
   ]

   def parse(self, response):
       hxs = HtmlXPathSelector(response)
       sites = hxs.select('//div[@id="col_3"]//div[@id="module3_1"]//div[@id="moduleData4952"]')
       items = []
       for site in sites:
           item = MlboddsItem()
           item['header'] = site.select('//div[@class="scoreboard-bar"]//h2//span[position()>1]//text()').extract()# | /*//table[position()<2]//tr//th[@colspan="2"]//text()').extract()
           item['game1'] = site.select('/*//table[position()=1]//tr//td[@class="tbl-odds-c2"]//text() | /*//table[position()=1]//tr//td[@class="tbl-odds-c4"]//text() | /*//table[position()=1]//tr//td[@class="tbl-odds-c6"]//text()').extract()
           items.append(item)
       return items

The results are returned in a random order, for example it returns the 29th, then the 28th, then the 30th. I've tried changing the scheduler order from DFO to BFO, just in case that was the problem, but that didn't change anything.

scrapy hashmap asynchronous sorting python
13 Answers

There is a much easier way to make scrapy follow the order of starts_url: you can just uncomment and change the concurrent requests in settings.py to 1.

Configure maximum concurrent requests performed by Scrapy (default: 16) 
CONCURRENT_REQUESTS = 1
user_93233970

I know this is an old question but I struggled with this problem today and was not completely satisfied with the solutions I found in this thread. Here's how I handled it.

the spider:

import scrapy

class MySpider(scrapy.Spider):

    name = "mySpider"

    start_urls = None

    def parse(self, response):
        #your parsing code goes here

    def __init__(self, urls):
        self.start_urls = urls

and the spider runner:

from twisted.internet import reactor, defer
import spiders.mySpider as ms
from scrapy.crawler import CrawlerRunner

urls = [
    'http://address1.com',
    'http://address2.com',
    'http://address3.com'
   ]

runner = CrawlerRunner()

@defer.inlineCallbacks
def crawl():
    for url in urls:
        yield runner.crawl(ms.MySpider, urls = [url])
    reactor.stop()

crawl()
reactor.run()

this code calls the spider with a url from the list passed as a parameter and then waits until it is finished before calling the spider again with the next url

user_35876210

Most of answers suggest passing urls one by one or limiting the concurrency to 1, which will slow you down significantly if you're scraping multiple urls.

While I faced this same problem my solution was using the callback arguments to store scraped data from all the urls, and sort it using the order of the initial urls, then return all the scraped data in ordered at once, something like this:

import scrapy

class MLBoddsSpider(scrapy.Spider):
   name = "sbrforum.com"
   allowed_domains = ["sbrforum.com"]
   to_scrape_urls = [
       "http://www.sbrforum.com/mlb-baseball/odds-scores/20110328/",
       "http://www.sbrforum.com/mlb-baseball/odds-scores/20110329/",
       "http://www.sbrforum.com/mlb-baseball/odds-scores/20110330/"
   ]

   def start_requests(self):
       data = {}
       for url in self.to_scrape_urls:
           yield scrapy.Request(url, self.parse, cb_kwargs=data)

   def parse(self, response, **kwargs):
       # scrape the data and add it to kwargs
       kwargs[response.url] = response.css('myData').get()

       # check if all urls has been scraped yet
       if len(kwargs) == len(self.to_scrape_urls):
           # return a sorted list of your data
           return [kwargs[url] for url in self.to_scrape_urls]
user_112379380

add this in settings

SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue' 
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'
user_172910370
Related