I am writing a crawler to crawl some forum contents and all my HTTP connection is using Apache Http Client. As suggested by the official documentation, I'm using a single Http client for a single forum server and this client, equipped with a PoolingHttpClientConnectionManager instance, can execute multiple requests by multiple execution threads at the same time.
One important attribute of this Pooling connection manager is maximum number of connections per route (which is 2 by default). I am confused which is the optimal (general) limit for this that ensure the speed of crawling but not overload the server?
(By general, I meant an average number that work for a general forum server in different cases cause I will set it static when I initialize the connection manager).
Besides that, I would really appreciate if someone know how to dynamically manage the limit per route based on server feedback in HttpClient 4.5 or other similar library.
Thanks really much for helping!