Channel hangs, probably not closing at the right place

Issue

I’m trying to learn Go while writing a small program. The program should parse a PATH recursivelys as efficient and fast as possible and output the full filename (with the path included) and the sha256 file hash of the file.

If the file hashing generates fails, I wanna keep the error and add it to the string (at the hash position).

The result should return a string on the console like:
fileXYZ||hash

Unfortunately, the programs hangs at some point. I guess some of my channels are not closing properly and waiting indefinitely for input. I’ve been trying for quite some time to fix the problem, but without success.

Does anyone have an idea why the output hangs? Many many thx in advance, any input/advice for a Go newcomer is welcome too ;-).

(I wrote separate functions as I wanna add additional features after having fixed this issue.)

Thanks a lot!
Didier

Here is the code:

import (
    "crypto/sha256"
    "encoding/hex"
    "flag"
    "fmt"
    "io"
    "log"
    "os"
    "path/filepath"
    "time"
)

func main() {
    pathParam := flag.String("path", ".", "Enter Filesystem Path to list folders")
    flag.Parse()
    start := time.Now()
    run(*pathParam)
    elapsed := time.Since(start)
    log.Printf("Time elapsed: %v", elapsed)
}

func run(path string) {
    chashes := make(chan string, 50)
    cfiles := make(chan string)

    go func() {
        readfs(path, cfiles)
        defer close(cfiles)
    }()
    go func() {
        generateHash(cfiles, chashes)
    }()
    defer close(chashes)
    for hash := range chashes {
        fmt.Println(hash)
    }
}

func readfs(path string, cfiles chan string) {
    files, err := os.ReadDir(path)
    if err != nil {
        log.Fatalln(err)
    }
    for _, file := range files {
        filename := filepath.Join(path, file.Name())
        if file.IsDir() {
            readfs(filename, cfiles)
            continue
        } else {
            cfiles <- filename
        }
    }
}

func generateHash(cfiles chan string, chashes chan string) {
    for filename := range cfiles {
        go func(filename string) {
            var checksum string
            var oError bool = false
            file, err := os.Open(filename)
            if err != nil {
                oError = true
                errorMsg := "ERROR: " + err.Error()
                log.Println(errorMsg)
                checksum = errorMsg
            }
            defer file.Close()

            if !oError {
                hash := sha256.New()
                if _, err := io.Copy(hash, file); err != nil {
                    errorMsg := "ERROR: " + err.Error()
                    log.Println(errorMsg)
                    checksum = errorMsg
                }
                if len(checksum) == 0 {
                    checksum = hex.EncodeToString(hash.Sum(nil))
                }
            }
            chashes <- filename + "||" + checksum
        }(filename)
    } //for files
}

Solution

The following loop hangs because chashes is not closed.

for hash := range chashes {
    fmt.Println(hash)
}

Fix by closing chashes after all the hashers are completed. Use a sync.WaitGroup to wait for the hashers to complete.

func generateHash(cfiles chan string, chashes chan string) {
    var wg sync.WaitGroup
    for filename := range cfiles {
        wg.Add(1)
        go func(filename string) {
            defer wg.Done()
            var checksum string
            var oError bool = false
            file, err := os.Open(filename)
            if err != nil {
                oError = true
                errorMsg := "ERROR: " + err.Error()
                log.Println(errorMsg)
                checksum = errorMsg
            }
            defer file.Close()

            if !oError {
                hash := sha256.New()
                if _, err := io.Copy(hash, file); err != nil {
                    errorMsg := "ERROR: " + err.Error()
                    log.Println(errorMsg)
                    checksum = errorMsg
                }
                if len(checksum) == 0 {
                    checksum = hex.EncodeToString(hash.Sum(nil))
                }
            }
            chashes <- filename + "||" + checksum
        }(filename)
    } //for files

    // Wait for the hashers to complete.
    wg.Wait()

    // Close the channel to cause main() to break
    // out of for range on chashes.
    close(chashes)
}

Remove defer close(chashes) from run().

Run an example on the Go playground.

Answered By – Bayta Darell

Answer Checked By – Jay B. (GoLangFix Admin)

Leave a Reply

Your email address will not be published.