What are the best configurations for crawling millions of hosts with colly and golang?

Viewed 26

I am sort of a newbie at Go and I am trying to implement a crawler that will send exactly 1 get request to each host and save the body, headers and some more info to my database.

I have already tried implementing my own crawlers with net/http and fasthttp with a worker pool, a semaphore approach and colly. But the performance is not as fast as desired. I might need to do some extra configurations for the crawling but I don't know them. For know I disable keep-alives, set InsecureSkipVerify: true, MaxIdleConnsPerHost: 0, Timeout: 15 secs. Other than this I am trying to use the http and fasthttp clients according to their recommendations (reuse clients, acquire requests, etc.).

Here are my colly configurations:

    c.MaxDepth = 1
    config := &tls.Config{
        InsecureSkipVerify: true,
    }
    c.WithTransport(&http.Transport{
        DisableKeepAlives: true,
        TLSClientConfig:   config,
        DialContext: (&net.Dialer{
            Timeout: crawlTimeout,
        }).DialContext,
        MaxIdleConns:          5,
        IdleConnTimeout:       crawlTimeout,
        TLSHandshakeTimeout:   crawlTimeout,
        ExpectContinueTimeout: 1 * time.Second,
    })

I have benchmarked my current implementations and here are my current results for crawling 500 hosts with 200 workers in each implementation.

BenchmarkRunCollyCrawler-4                     1        10160866756 ns/op
BenchmarkRunSemaphoreBDC-4                     1        6331642362 ns/op
BenchmarkRunSemaphoreBDCWithHttp-4             1        9205261387 ns/op
BenchmarkRunBDC-4                              1        75540165775 ns/op
BenchmarkRunBDCWithHttp-4                      1        37692119211 ns/op

What are the configurations for speeding up my crawlers with fasthttp, net/http, and colly?

0 Answers
Related