I'd like to track execution of some long running process and show the user completion percentage and errors (if any). If it's one long running process, then it's easy - you can create channels for progress (percentage) and error. What would the correct way to implement such logic when we have X long running processes?
Below is a snippet of code that works, but I don't really like how it's implemented.
I created a struct ProgressTracker that keeps Url (as a field), Error, Progress
as channels. I keep such ProgressTracker in a slice and once I submit all tasks I iterate via the slice of ProgressTracker and listen to channels for each tracker in ProgressTracker. Once the number of submitted requests == number of received responses - exit the loop.
Is it Go idiomatic solution? It would be easier to pass ProgressTracker to the function as a channel, but I don't know how to properly send "progress", "error" and "complete" events in such case.
The code is below, the same is available in Go playground: https://go.dev/play/p/f3hXJsZR9WV
package main
import (
"errors"
"fmt"
"strings"
"sync"
"time"
)
type ProgressTracker struct {
Progress chan int
Error chan error
Complete chan bool
Url string
}
/**
This method sleeps for 1 second and sends progress (in %) in each iteration to Progress channel
For .net sites on 3rd iteration fail with error
When everything is completed, send a message to Complete channel
*/
func work(url string, tracker *ProgressTracker) {
tracker.Url = url
fmt.Printf("processing url %s\n", url)
for i := 1; i <= 5; i++ {
time.Sleep(time.Second)
if i == 3 && strings.HasSuffix(url, ".net") {
tracker.Error <- errors.New("emulating error for .net sites")
tracker.Complete <- true
}
progress := 20 * i
tracker.Progress <- progress
}
tracker.Complete <- true
}
func main() {
var trackers []*ProgressTracker
var urls = []string{"google.com", "youtube.com", "someurl.net"}
var wg sync.WaitGroup
wg.Add(len(urls))
for _, url := range urls {
tracker := &ProgressTracker{
Progress: make(chan int),
Error: make(chan error),
Complete: make(chan bool),
}
trackers = append(trackers, tracker)
go func(workUrl string, progressTracker *ProgressTracker) {
work(workUrl, progressTracker)
}(url, tracker)
}
go func() {
wg.Wait()
}()
var processed = 0
//iterate through all trackers and select each channel.
//Exit from this loop when number of processed requests equals the number of trackers
for {
for _, t := range trackers {
select {
case pr := <-t.Progress:
fmt.Printf("Url = %s, progress = %d\n", t.Url, pr)
case err := <-t.Error:
fmt.Printf("Url = %s, error = %s\n", t.Url, err.Error())
case <-t.Complete:
fmt.Printf("Url = %s is completed\n", t.Url)
processed = processed + 1
if processed == len(trackers) {
fmt.Printf("Everything is completed, exit")
return
}
}
}
}
}
UPD: If I add a delay to one of the tasks, then the for loop where I select all the channels will also wait for the slowest worker on each iteration. Go playground: https://go.dev/play/p/9FvDE7ZGIrP Updated work function:
func work(url string, tracker *ProgressTracker) {
tracker.Url = url
fmt.Printf("processing url %s\n", url)
for i := 1; i <= 5; i++ {
if url == "google.com" {
time.Sleep(time.Second * 3)
}
time.Sleep(time.Second)
if i == 3 && strings.HasSuffix(url, ".net") {
tracker.Error <- errors.New("emulating error for .net sites")
tracker.Complete <- true
return
}
progress := 20 * i
tracker.Progress <- progress
}
tracker.Complete <- true
}