In Golang http.client, why the increased pool size doesn't result in better performance?

Viewed 48

I noticed that in Golang http.client, there is connections pool settings, like: MaxIdleConns or DefaultMaxIdleConnsPerHost, check the source code here.

I will expect that with the increase idle connections pool, the performance will be improved dramatically.

So I wrote a simple program try to proof that, but unfortunately, it shows that the pool size is not relevant to the performance at all. Could anyone help to explain? Also I post the program code below, please also check if there is an error there? Thank you!

// In this example, there are 3 routers:
// * /per-client-per-request: this route a new http.Client instance for each request.
// * /2pool: this route uses the default transport which has 2 maxIdlConnsPerHost 
// * /100pool: this route uses a modified transport which has 100 maxIdlConnsPerHost
// Ideally, the /per-client-per-request should has the worst performance, /2pool is better, and /100pool should be the best.
// However, the result is totally random. Sometimes /2pool is the best, sometimes /100pool is the best, and sometimes /per-client-per-request is the best.
package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "os"

    "github.com/gorilla/mux"
)

func startResourceServer() {
    r := mux.NewRouter()
    r.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        // time.Sleep(5 * time.Millisecond)
        w.Write([]byte("Hello World"))
    })
    srv := &http.Server{
        Addr:    "0.0.0.0:8081",
        Handler: r,
    }
    go func() {
        if err := srv.ListenAndServe(); err != nil {
            log.Println(err)
        }
    }()
}

type Handler struct {
    httpClientWith2Pool   *http.Client
    httpClientWith100Pool *http.Client
}

func main() {
    startResourceServer()

    transportWithMax100Pool := http.DefaultTransport.(*http.Transport).Clone()
    transportWithMax100Pool.MaxIdleConnsPerHost = 100
    handler := &Handler{
        httpClientWith100Pool: &http.Client{Transport: transportWithMax100Pool},
        httpClientWith2Pool:   &http.Client{},
    }
  
    r := mux.NewRouter()
    r.HandleFunc("/2pool", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte(fmt.Sprintf("res len: %d", visitRemoteServer(handler.httpClientWith2Pool))))
    })
    r.HandleFunc("/100pool", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte(fmt.Sprintf("res len: %d", visitRemoteServer(handler.httpClientWith100Pool))))
    })
    r.HandleFunc("/per-client-per-request", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte(fmt.Sprintf("res len: %d", visitRemoteServer(&http.Client{}))))
    })
    srv := &http.Server{
        Addr:    "0.0.0.0:8080",
        Handler: r,
    }
    log.Fatal(srv.ListenAndServe())
}

func visitRemoteServer(httpClient *http.Client) int {
    res, err := httpClient.Get("http://localhost:8081")
    if err != nil {
        fmt.Printf("error making http request: %s", err)
        os.Exit(1)
    }
    body, _ := io.ReadAll(res.Body)
    return len(body)
}

And I use vegeta tool to test:

echo "GET http://localhost:8080/per-client-per-request" | vegeta attack -duration=5s -rate=200 | vegeta report
echo "GET http://localhost:8080/2pool" | vegeta attack -duration=5s -rate=200 | vegeta report
echo "GET http://localhost:8080/100pool" | vegeta attack -duration=5s -rate=200 | vegeta report

I expected that the results should gets better and better, but unfortunately, it is very random, and even the last load test which has 100 connections pool usually was the worst case. Check the below typical result:

Requests      [total, rate, throughput]         1000, 200.22, 200.18
Duration      [total, attack, wait]             4.995s, 4.995s, 857.5µs
Latencies     [min, mean, 50, 90, 95, 99, max]  206.583µs, 833.203µs, 599.667µs, 1.577ms, 2.167ms, 3.871ms, 8.396ms
Bytes In      [total, mean]                     11000, 11.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:1000  
Error Set:

Requests      [total, rate, throughput]         1000, 200.21, 200.17
Duration      [total, attack, wait]             4.996s, 4.995s, 1.004ms
Latencies     [min, mean, 50, 90, 95, 99, max]  258.166µs, 862.518µs, 700.689µs, 1.505ms, 2.052ms, 3.312ms, 7.735ms
Bytes In      [total, mean]                     11000, 11.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:1000  
Error Set:

Requests      [total, rate, throughput]         1000, 200.20, 200.18
Duration      [total, attack, wait]             4.996s, 4.995s, 594.958µs
Latencies     [min, mean, 50, 90, 95, 99, max]  238µs, 916.789µs, 698.426µs, 1.542ms, 2.252ms, 3.896ms, 9.544ms
Bytes In      [total, mean]                     11000, 11.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:1000  
Error Set:

Update #1

Based on the @steffen-ullrich comment, I changed my program as below. The new code will use http.Client directly send request to google. Two cases are using maxIdleConnectionsPerHost to 2 and 100 respectively.

package main

import (
    "fmt"
    "net/http"
    "sync"
    "time"
)

const TIMES = 300

var wg sync.WaitGroup

func check(maxIdleConnsPerHost int) {
    transport := http.DefaultTransport.(*http.Transport).Clone()
    transport.MaxIdleConns = 100
    transport.MaxIdleConnsPerHost = maxIdleConnsPerHost
    c := make(chan int)
    httpClient := &http.Client{Transport: transport}
    results := make([]int, 0)

    go func() {
        for i := 0; i < TIMES; i++ {
            c <- i
            time.Sleep(10 * time.Millisecond)
        }
        close(c)
    }()

    wg.Add(TIMES)

    for range c {
        go func(httpClient *http.Client) {
            start := time.Now()
            defer wg.Done()
            defer func() {
                results = append(results, int(time.Since(start).Milliseconds()))
            }()
            _, err := httpClient.Get("https://www.google.com")
            if err != nil {
                panic(err)
            }
        }(httpClient)
    }

    wg.Wait()

    sum := 0
    for _, v := range results {
        sum += v
    }

    // fmt.Printf("list: %v \n", results)
    fmt.Printf("When maxIdleConnectionsPerHost is %d, ", maxIdleConnsPerHost)
    fmt.Printf("Average elapse ms: %d \n", sum/TIMES)
}

func main() {
    check(2)
    check(100)
}

However, the result still showing the connection pool size is irrelevant to the performance. The below is the results I ran 4 times.

When maxIdleConnectionsPerHost is 2, Average elapse ms: 649 
When maxIdleConnectionsPerHost is 100, Average elapse ms: 772 

When maxIdleConnectionsPerHost is 2, Average elapse ms: 755 
When maxIdleConnectionsPerHost is 100, Average elapse ms: 739 

When maxIdleConnectionsPerHost is 2, Average elapse ms: 748 
When maxIdleConnectionsPerHost is 100, Average elapse ms: 756 

When maxIdleConnectionsPerHost is 2, Average elapse ms: 810 
When maxIdleConnectionsPerHost is 100, Average elapse ms: 780 
0 Answers
Related