Is Scrapy Asychronous by Default?

user_12857878 0
Viewed 2806

I recently ran a spider in my project but I feel like scrapy it is waiting until one page is finished to move on the other one. if I am correct in scrapy's nature it moves to another page until the previous one's response is received. in this page after scrolling down I saw async def used which means that method was explicitly made asynchronous by adding that. and If I don't put async-await in my spiders, won't they become asynchronous. do they wait until a response is received? please let me know If I have any misconceptions and Thank you in advance.

scrapy web-scraping asynchronous python
2 Answers

yes as @Gallaecio said scrapy is asynchronous by default. and I want to add that I could make the synchronous part of code asynchronous. by this,

synchronous request

def parse(self, response, current_page):
    url = 'https://example.com/search?page={}'
    # do some stuff
    self.current_page += 1
    yield Request(url.format(current_page), call_back=self.parse)

asynchronous request

def parse(self, response):
    # do some stuff
    for page in range(self.total_pages):
        yield Requests(f'https://example.com/search?page={page}', 
                          callback=self.parse)
user_128578780

Scrapy is asynchronous by default.

Using coroutine syntax, introduced in Scrapy 2.0, simply allows for a simpler syntax when using Twisted Deferreds, which are not needed in most use cases, as Scrapy makes its usage transparent whenever possible.

The only reason why your spiders may seem synchronous would be because you only yield a new Request object from the callback of a previous request. If you yield multiple requests from start_requests, or have multiple URLs in start_urls, those will be handled asynchronously, according to your concurrency settings (Scrapy’s default is 8 concurrent requests per domain, 16 total).

user_9393640
Related