I'm using Scrapy on my dedicated server, I would like to know how get the best performance for my crawler.
Here is my custom settings :
custom_settings = {
'RETRY_ENABLED': True,
'DEPTH_LIMIT' : 0,
'DEPTH_PRIORITY' : 1,
'LOG_ENABLED' : False,
'CONCURRENT_REQUESTS_PER_DOMAIN' : 32,
'CONCURRENT_REQUESTS' : 64,
}
I actually crawl around 200 link/minutes.
Server :
32 Go RAM : DDR4 ECC 2133 MHz
CPU : 4c/8t : 2,2 / 2,6 GHz