I am sort of a newbie at Go and I am trying to implement a crawler that will send exactly 1 get request to each host and save the body, headers and some more info to my database.
I have already tried implementing my own crawlers with net/http and fasthttp with a worker pool, a semaphore approach and colly. But the performance is not as fast as desired. I might need to do some extra configurations for the crawling but I don't know them. For know I disable keep-alives, set InsecureSkipVerify: true, MaxIdleConnsPerHost: 0, Timeout: 15 secs. Other than this I am trying to use the http and fasthttp clients according to their recommendations (reuse clients, acquire requests, etc.).
Here are my colly configurations:
c.MaxDepth = 1
config := &tls.Config{
InsecureSkipVerify: true,
}
c.WithTransport(&http.Transport{
DisableKeepAlives: true,
TLSClientConfig: config,
DialContext: (&net.Dialer{
Timeout: crawlTimeout,
}).DialContext,
MaxIdleConns: 5,
IdleConnTimeout: crawlTimeout,
TLSHandshakeTimeout: crawlTimeout,
ExpectContinueTimeout: 1 * time.Second,
})
I have benchmarked my current implementations and here are my current results for crawling 500 hosts with 200 workers in each implementation.
BenchmarkRunCollyCrawler-4 1 10160866756 ns/op
BenchmarkRunSemaphoreBDC-4 1 6331642362 ns/op
BenchmarkRunSemaphoreBDCWithHttp-4 1 9205261387 ns/op
BenchmarkRunBDC-4 1 75540165775 ns/op
BenchmarkRunBDCWithHttp-4 1 37692119211 ns/op
What are the configurations for speeding up my crawlers with fasthttp, net/http, and colly?