I'm writing a spider for scraping data about cars from a carsharing websites https://fr.be.getaround.com/. The objective is to divide my spider in two parts. First, it scrapes data for available cars and keeps unavailable cars aside. Second, once all information about available cars is scraped, thus at the end of the process, the spider scrapes additional information for unavailable cars. For this second part, I've added the spider_idle method in my spider. Doing so, it should be called once no available cars is remaining in the waiting list. However, I've added a DOWNLOAD_DELAY (5 seconds) and Autothrottle is enabled. I was wondering, do spider_idle will be called during the waiting time between each request (within the 5 seconds) ?