How to tune the number of goroutines used to download N files?

Issue

I’m making a Golang program which needs to download N files and then perform some task on each one. In particular there is a pipeline of operation to do on each single SWF file:

  1. Make an http get request to get the file
  2. Save it locally
  3. Convert it in PNG

Doing it sequentially could be very inefficient. Doing it with N goroutines could also be non-optimal.

How do I choose / limit the number of goroutines?

Solution

A goroutine is lightweight by design:

A goroutine has a simple model: it is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.

The limit is not in the number of goroutines you generate, but in what they do.

In your case, opening files (to write a downloaded stream) concurrently could meet the limit of number of opened files per process or in general

Answered By – VonC

Answer Checked By – Mildred Charles (GoLangFix Admin)

Leave a Reply

Your email address will not be published.