DEV Community

Cover image for Processing File concurrently with using Goroutines
Sandilya Karavadi
Sandilya Karavadi

Posted on

Processing File concurrently with using Goroutines

#go

Problem

  • How to Read File
  • How to Process File (i.e Channel) Concurrently with Goroutines

Solution

Reading a File

  • Spin up a goroutine to read data from file in chunks using 
scanner.Scan()
Enter fullscreen mode Exit fullscreen mode

Image description

Processing Channel

  • Spin Up N goroutine
  • Each one will take a Chunk from the Channel
  • Extract data from the Chunk and execute processFn 
func ProcessFile(file io.Reader, numWorkers int, chunkSize int, processFn func(data []byte))
Enter fullscreen mode Exit fullscreen mode

Image description

Github Repo for code

Github Repo

Inspired by

Top comments (0)