I'm trying to scrape homegate.ch. It's a web site with some sort of bot detection system. I integrated a several functions to my spider to deploy some 16.000 user agents. As a proxy, I use Leafproxy's residential proxies. I imported each randomly generated proxy in each Request's meta as "proxy" key. I submitted my proxies to my spider class and have a function that creates a proxy for each request. I also set DOWNLOAD_DELAY to 5 second and put the auto-throttle mode on
Problem: **My scraper receives 200 responses for listing page and following requestes by pagination also returns 200 requests. But the requests to property links (scraped from the listing page) returns 403 responses. Pagination works until page 20.
Interestingly, my proxies work for the requests to listing pages but to the item link it doesn't work. What could be the possible solution? This is why I'm hopeful that there could be a solution.
When I make the same requests on shell, the same happens. Listing pages return 200s but item links returns 403s.
I did a testing for the web page. Results say that it doesn't have the following bot detection systems: "Shape Security, DataDome, Distil, Imperva, Incapsula, PerimeterX, Akamai, FingerprintJS, FingerprintJS Pro, Kasada, WhiteOps, ShieldSquare, ThreatMetrix, F5, Cloudflare, Arkose Labs, Reblaze, Forter Protection, Meetrics Check, reCAPTCHA, generic fingerprinting & bot detection"
sample _listing _url = "https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ah=1000"
I can provide the full spider code if needed.
Settings:
ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 5
COOKIES_ENABLED = True
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 5
Request method for listing page (returning 200)
yield Request(
url = i,
callback = self.parse,
headers = {
"user-agent": random.choice(self.user_agents)['user_agent']
},
meta = {"proxy": proxy}
)
Failing request method for scraped property links:
yield Request(
url = son_link,
callback= self.item_parser,
headers = {
"user-agent": UA
},
meta = {"proxy": proxy}
)
Logs:
draco@draco:~/docs/scraping/scrapyyy/thomas$ scrapy crawl home
x5rhfuesar27hwcohtxu2x+CAKef4uI:x940jt4upr
2022-03-05 05:53:53 [scrapy.utils.log] INFO: Scrapy 2.5.1 started (bot: thomas)
2022-03-05 05:53:53 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.7.0, Python 3.8.10 (default, Nov 26 2021, 20:14:08) - [GCC 9.3.0], pyOpenSSL 22.0.0 (OpenSSL 1.1.1m 14 Dec 2021), cryptography 36.0.1, Platform Linux-5.13.0-30-generic-x86_64-with-glibc2.29
2022-03-05 05:53:53 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2022-03-05 05:53:53 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_ENABLED': True,
'BOT_NAME': 'thomas',
'DOWNLOAD_DELAY': 5,
'NEWSPIDER_MODULE': 'thomas.spiders',
'SPIDER_MODULES': ['thomas.spiders'],
'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
'(KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'}
2022-03-05 05:53:53 [scrapy.extensions.telnet] INFO: Telnet Password: 4c18c39ee1e60545
2022-03-05 05:53:53 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats',
'scrapy.extensions.throttle.AutoThrottle']
2022-03-05 05:53:58 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-03-05 05:53:58 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'thomas.middlewares.ThomasSpiderMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2022-03-05 05:53:58 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2022-03-05 05:53:58 [scrapy.core.engine] INFO: Spider opened
2022-03-05 05:53:58 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-03-05 05:53:58 [home] INFO: Spider opened: home
2022-03-05 05:53:58 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024
INITIAL REQUEST
UA:Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2661.102 Safari/537.36
2022-03-05 05:54:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ah=1000> (referer: None)
New Listing Page
2022-03-05 05:54:01 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.homegate.ch/buy/3001719592> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=2
2022-03-05 05:54:04 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001719592> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ah=1000)
2022-03-05 05:54:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001719592>: HTTP status code is not handled or not allowed
2022-03-05 05:54:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=2> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ah=1000)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=3
2022-03-05 05:54:14 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001654517> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=2)
2022-03-05 05:54:14 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001654517>: HTTP status code is not handled or not allowed
2022-03-05 05:54:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=3> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=2)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=4
2022-03-05 05:54:23 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001499305> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=3)
2022-03-05 05:54:24 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001499305>: HTTP status code is not handled or not allowed
2022-03-05 05:54:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=4> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=3)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=5
2022-03-05 05:54:36 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001708121> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=4)
2022-03-05 05:54:37 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001708121>: HTTP status code is not handled or not allowed
2022-03-05 05:54:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=5> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=4)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=6
2022-03-05 05:54:50 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001158904> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=5)
2022-03-05 05:54:51 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001158904>: HTTP status code is not handled or not allowed
2022-03-05 05:54:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=6> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=5)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=7
2022-03-05 05:54:58 [scrapy.extensions.logstats] INFO: Crawled 11 pages (at 11 pages/min), scraped 0 items (at 0 items/min)
2022-03-05 05:55:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=7> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=6)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=8
2022-03-05 05:55:10 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001183340> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=6)
2022-03-05 05:55:10 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001183340>: HTTP status code is not handled or not allowed
2022-03-05 05:55:14 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001721582> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=7)
2022-03-05 05:55:14 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001721582>: HTTP status code is not handled or not allowed
2022-03-05 05:55:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=8> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=7)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=9
2022-03-05 05:55:37 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001657070> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=8)
2022-03-05 05:55:37 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001657070>: HTTP status code is not handled or not allowed
2022-03-05 05:55:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=9> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=8)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=10
2022-03-05 05:55:51 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001660540> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=9)
2022-03-05 05:55:51 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001660540>: HTTP status code is not handled or not allowed
2022-03-05 05:55:58 [scrapy.extensions.logstats] INFO: Crawled 18 pages (at 7 pages/min), scraped 0 items (at 0 items/min)
2022-03-05 05:55:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=10> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=9)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=11
2022-03-05 05:56:05 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001475976> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=10)
2022-03-05 05:56:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001475976>: HTTP status code is not handled or not allowed
2022-03-05 05:56:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=11> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=10)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=12
2022-03-05 05:56:19 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001318386> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=11)
2022-03-05 05:56:19 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001318386>: HTTP status code is not handled or not allowed
2022-03-05 05:56:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=12> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=11)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=13
2022-03-05 05:56:33 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001574339> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=12)
2022-03-05 05:56:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001574339>: HTTP status code is not handled or not allowed
2022-03-05 05:56:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=13> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=12)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=14
2022-03-05 05:56:44 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001719528> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=13)
2022-03-05 05:56:44 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001719528>: HTTP status code is not handled or not allowed
2022-03-05 05:56:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=14> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=13)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=15
2022-03-05 05:56:56 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001722598> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=14)
2022-03-05 05:56:56 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001722598>: HTTP status code is not handled or not allowed
2022-03-05 05:56:58 [scrapy.extensions.logstats] INFO: Crawled 28 pages (at 10 pages/min), scraped 0 items (at 0 items/min)
2022-03-05 05:56:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=15> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=14)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=16
2022-03-05 05:57:03 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001474691> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=15)
2022-03-05 05:57:03 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001474691>: HTTP status code is not handled or not allowed
2022-03-05 05:57:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=16> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=15)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=17
2022-03-05 05:57:16 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001691729> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=16)
2022-03-05 05:57:16 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001691729>: HTTP status code is not handled or not allowed
2022-03-05 05:57:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=17> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=16)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=18
2022-03-05 05:57:28 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001486373> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=17)
2022-03-05 05:57:28 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001486373>: HTTP status code is not handled or not allowed
2022-03-05 05:57:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=18> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=17)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=19
2022-03-05 05:57:42 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001714917> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=18)
2022-03-05 05:57:42 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001714917>: HTTP status code is not handled or not allowed
2022-03-05 05:57:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=19> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=18)
New Listing Page
NEXT LISTING PAGE ===https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=20
2022-03-05 05:57:52 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001725111> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=19)
2022-03-05 05:57:52 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001725111>: HTTP status code is not handled or not allowed
2022-03-05 05:57:58 [scrapy.extensions.logstats] INFO: Crawled 38 pages (at 10 pages/min), scraped 0 items (at 0 items/min)
2022-03-05 05:58:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=20> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=19)
New Listing Page
2022-03-05 05:58:06 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.homegate.ch/buy/3001711682> (referer: https://www.homegate.ch/buy/apartment/canton-bern/matching-list?ep=20)
2022-03-05 05:58:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.homegate.ch/buy/3001711682>: HTTP status code is not handled or not allowed
2022-03-05 05:58:06 [scrapy.core.engine] INFO: Closing spider (finished)
2022-03-05 05:58:06 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 27766,
'downloader/request_count': 40,
'downloader/request_method_count/GET': 40,
'downloader/response_bytes': 1192572,
'downloader/response_count': 40,
'downloader/response_status_count/200': 20,
'downloader/response_status_count/403': 20,
'dupefilter/filtered': 361,
'elapsed_time_seconds': 248.533228,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2022, 3, 5, 2, 58, 6, 604154),
'httpcompression/response_bytes': 4715026,
'httpcompression/response_count': 40,
'httperror/response_ignored_count': 20,
'httperror/response_ignored_status_count/403': 20,
'log_count/DEBUG': 41,
'log_count/INFO': 35,
'memusage/max': 102510592,
'memusage/startup': 89325568,
'request_depth_max': 20,
'response_received_count': 40,
'scheduler/dequeued': 40,
'scheduler/dequeued/memory': 40,
'scheduler/enqueued': 40,
'scheduler/enqueued/memory': 40,
'start_time': datetime.datetime(2022, 3, 5, 2, 53, 58, 70926)}
2022-03-05 05:58:06 [scrapy.core.engine] INFO: Spider closed (finished)