Golang Concurrency With HTTP Requests

Question

Golang Concurrency With HTTP Requests Giving Error After Running Some Times. I Am not understanding what is causing this issue. I am

Error: 2022/08/15 15:32:11 API ErrorPost "https://[redacted]/checknum": read tcp [redacted]:63062->[redacted]:443: wsarecv: An existing connection was forcibly closed by the remote host.

Code:

package main

import (
    "bytes"
    "crypto/tls"
    "fmt"
    "io/ioutil"
    "bufio"
    "net/http"
    "strings"
    "sync"
    "os"
    "log"
)

func checknum(wg *sync.WaitGroup, number string) {
    defer wg.Done()
    headers := map[string]string{
        "User-Agent":   "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36",
        "Content-Type": "application/json; charset=UTF-8",
    }
    var data = []byte("\"number\": \""+number+"\"}")
    httpRequest("https://[redacted]/checknum", "POST", data, headers, number)
}

func httpRequest(targetUrl string, method string, data []byte, headers map[string]string, number string) {
    request, error := http.NewRequest(method, targetUrl, bytes.NewBuffer(data))
    for k, v := range headers {
        request.Header.Set(k, v)
    }
    customTransport := &http.Transport{
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
        DisableKeepAlives: true,
    }

    request.Close = true

    client := &http.Client{Transport: customTransport}
    response, error := client.Do(request)

    if response != nil {
        defer response.Body.Close()
    }

    if error != nil {
        log.Fatal("API Error ",error)
    }

    body, err := ioutil.ReadAll(response.Body)

    if err != nil {
        log.Fatal("Body Error ",error)
    }
    if strings.Contains(string(body), `VALID`) {
        fmt.Println("[+]:",number)
    } else {
        fmt.Println("[-]:",number)
    }
}

func main() {
    var wg sync.WaitGroup
    file, err := os.Open(os.Args[1])
    if err != nil {
        fmt.Println("cannot able to read the file", err)
        return
    }
    defer file.Close()
    scanner := bufio.NewScanner(file)

    for scanner.Scan() {
        num := strings.TrimSpace(scanner.Text())
        if num != "" {
            wg.Add(1)
            go checknum(&wg, num)
        }
    }
    wg.Wait()
}

Note: My File Contains Almost 10000 Lines.

user_16998 · Answer

The answer is right there in your question.

wsarecv: An existing connection was forcibly closed by the remote host.

Your webserver is closing connections on you, likely because you are overwhelming it.

If you have access to this server, then check the logs. If you do not own this server then you should be acting like a better netizen and limiting the number of concurrent connections to the server.

You are declaring a custom transport for every call. It would be better to declare this once and use it for all connections.

If you do this, then you may benefit from http2, assuming the server supports it, and connections can be reused, shared, and multiplexed.

Golang Concurrency With HTTP Requests

1 Answers