Best performance for Scrapy

Viewed 3561

I'm using Scrapy on my dedicated server, I would like to know how get the best performance for my crawler.

Here is my custom settings :

custom_settings = {
    'RETRY_ENABLED': True,
    'DEPTH_LIMIT' : 0,
    'DEPTH_PRIORITY' : 1,
    'LOG_ENABLED' : False,
    'CONCURRENT_REQUESTS_PER_DOMAIN' : 32,
    'CONCURRENT_REQUESTS' : 64,
}

I actually crawl around 200 link/minutes.

Server :

32 Go RAM : DDR4 ECC 2133 MHz
CPU : 4c/8t : 2,2 / 2,6 GHz
2 Answers
Related