Concurrently processing several files in Go

Question

I have a system that receives hundreds of files per day for processing. I am relatively new to GO and was looking at building something to concurrently process the files. The workflow looks like this: Get list of files, for each file in list ->check file size->check file hash->Process file->Add processed file name and processing time to database-> Send file to destination. The process happens sequentially but I would like to process the files concurrently or in a fanout pattern.

Right now I have something that looks like the following:

err := filepath.Walk("/uploads", func(path string, info os.FileInfo, err error) error {
    filesToTransfer = append(filesToTransfer, path)
    return nil
})

if err != nil {
    ErrorLogger.Println(err)
}


for _, uploadedFile := range filesToTransfer {
   checkFileSize(uploadedFile.Name(), expectedSize)
   checkFileHash(uploadedFile.Name())
   readFileData(uploadedFile.Name())
   addProcessedToDatabase(uploadedFile.Name())
   sendFile(uploadedFile.Name())        
}

Would wrapping the 5 functions in an anonymous goroutine function accomplish what I am trying to do? Something like this:

err := filepath.Walk("/uploads", func(path string, info os.FileInfo, err error) error {
    filesToTransfer = append(filesToTransfer, path)
    return nil
})

if err != nil {
    ErrorLogger.Println(err)
}
  var wg WaitGroup

  for _, fileToTransfer := range filesToTransfer {
     wg.Add(1)
     go func(uploadFile string) {
         checkFileSize(uploadFile, expectedSize)
         checkFileHash(uploadedFile)
         readFileData(uploadedFile)
         addProcessedToDatabase(uploadedFile)
         sendFile(uploadedFile)   
         wg.Done()
     }(fileToTransfer)   
   wg.Wait()  
}

Edit: I modified the way I am gathering files in the directory. I could not figure out how to get an fs.FileInfo type to pass to my goroutine and made the correction pointed out in the comments.

Concurrently processing several files in Go

0 Answers