Using goroutines to iterate through file indefinitely

user_15585901 0
Viewed 199

I'm new to Go so please excuse my ignorance. I'm attempting to iterate through a bunch of wordlists line by line indefinitely with goroutines. But when trying to do so, it does not iterate or stops half way through. How would I go about this in the proper manner without breaking the flow?

package main

import (
    "bufio"
    "fmt"
    "os"
)


var file, _ = os.Open("wordlist.txt")
func start() {
    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
       fmt.Println(scanner.Text())
    }

}

func main(){
    for t := 0; t < 150; t++ {
        go start()
        fmt.Scanln()
    }
}

Thank you!

goroutine go iteration file loops
1 Answers

You declare file as a global variable. Sharing read/write file state amongst multiple goroutines is a data race and will give you undefined results.

Most likely, reads start where the last read from any of the goroutines left off. If that's end-of-file, it likely continues to be end-of-file. But, since the results are undefined, that's not guaranteed. Your erratic results are due to undefined behavior.

Here's a revised version of your program that declares a local file variable and uses a sync.Waitgroup to synchronize the completion of all the go start() goroutines and the main goroutine. The program checks for errors.

package main

import (
    "bufio"
    "fmt"
    "os"
    "sync"
)

func start(filename string, wg *sync.WaitGroup, t int) {
    defer wg.Done()

    file, err := os.Open(filename)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    lines := 0
    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        lines++
    }
    if err := scanner.Err(); err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(t, lines)
}

func main() {
    wg := &sync.WaitGroup{}
    filename := "wordlist.txt"
    for t := 0; t < 150; t++ {
        wg.Add(1)
        go start(filename, wg, t)
    }
    wg.Wait()
}
user_164658020
Related