Synchronize Goroutines with `sync.WaitGroup`

1. The Problem: Uncoordinated Goroutines

When you launch goroutines, they run independently of the function that started them. This means the main function, or any parent goroutine, will not automatically wait for them to finish. This often leads to a common bug where the main program exits before the goroutines have had a chance to complete their work.

Let's see this problem in action.

As you saw, the message from the goroutine was never printed because the main function didn't wait. In a real-world DevOps tool, this could mean your script reports success while all its background jobs failed to even run.

2. The Solution: sync.WaitGroup

A sync.WaitGroup is essentially a counter that blocks a goroutine's execution until the counter's value is zero. It provides a simple way to wait for a collection of goroutines to finish.

It has three core methods:

• Add(delta int): Increments the counter by delta. You call this to specify how many goroutines you are about to launch and need to wait for.
• Done(): Decrements the counter by one. Each goroutine calls this method when it finishes its work. It's just a shorthand for Add(-1).
• Wait(): Blocks the execution of the goroutine that calls it until the counter becomes zero. This is typically called in the main (or parent) goroutine.

The following diagram illustrates this flow:

Now, let's see how these methods are used to fix the problem from the first video.

Notice the use of defer wg.Done(). This is a crucial best practice that ensures the WaitGroup counter is decremented even if the function encounters an error or panics. We'll discuss this and other best practices more later.

3. Practical Use Case: Parallel Health Checks

Now for a more practical example that's highly relevant in a DevOps context. Imagine you need to build a tool that checks the status of several websites or API endpoints concurrently. Doing this sequentially would be slow, as you'd wait for each request to complete before starting the next. With goroutines and a WaitGroup, you can perform all checks in parallel and wait for all results.

This next video segment builds a small REST API that does exactly that.

4. Another Perspective: Aggregating Data for Performance

Another common use case is aggregating data from multiple sources to serve a single request. For example, fetching user profile details from one service and their activity from another. A WaitGroup allows you to parallelize these data-fetching operations, significantly reducing response time.

The following video does an excellent job of demonstrating the performance benefit. It starts with a slow, synchronous implementation and refactors it to use goroutines and a WaitGroup.

This before-and-after comparison clearly shows why concurrent execution coordinated by a WaitGroup is such a powerful pattern.

5. Best Practices and Common Pitfalls

While WaitGroup is simple, there are a few common mistakes that can lead to bugs like race conditions or deadlocks. Understanding these best practices is essential for using it correctly.

To reinforce these concepts, especially the danger of deadlocks, another resource provides clear examples of what can go wrong.

package main

import (
	"fmt"
	"sync"
	"time"
)

func processFile(filename string, wg *sync.WaitGroup) {
	defer wg.Done()
	fmt.Println("Processing", filename)
	time.Sleep(100 * time.Millisecond) // Simulate work
}

func main() {
	var wg sync.WaitGroup
	files := []string{"config1.yml", "config2.yml", "config3.yml"}

	for _, file := range files {
		go func(f string) {
			wg.Add(1) // Where is the problem?
			processFile(f, &wg)
		}(file)
	}

	wg.Wait()
	fmt.Println("All files processed.")
}

package main

import (
	"fmt"
	"sync"
	"time"
)

func processFile(filename string, wg *sync.WaitGroup) {
	defer wg.Done()
	fmt.Println("Processing", filename)
	time.Sleep(100 * time.Millisecond) // Simulate work
}

func main() {
	var wg sync.WaitGroup
	files := []string{"config1.yml", "config2.yml", "config3.yml"}

	for _, file := range files {
		wg.Add(1) // Correct: Increment the counter before starting the goroutine
		go func(f string) {
			processFile(f, &wg)
		}(file)
	}

	wg.Wait()
	fmt.Println("All files processed.")
}

Conclusion

In this lesson, you've learned how to use sync.WaitGroup to coordinate the completion of multiple goroutines. This is a fundamental pattern for parallelizing tasks, improving performance, and building efficient DevOps tools in Go.

Key Takeaways:

• sync.WaitGroup is a simple counter used to wait for a group of goroutines to finish.
• The main pattern involves Add() to set the number of tasks, launching goroutines that each call Done() upon completion, and using Wait() to block until all tasks are finished.
• Always call Add() before launching the associated goroutine to prevent a race condition.
• Always use defer wg.Done() at the beginning of your goroutine's work function to ensure the counter is decremented, even in case of panics.
• WaitGroup is ideal when you need to "fire and forget" a batch of tasks and just wait for them all to complete, without needing to pass data back from them.

Next Up

WaitGroup is excellent for synchronizing the completion of tasks. However, what happens when those concurrent tasks need to access and modify the same shared data, like a map or a slice? If multiple goroutines write to the same memory location at the same time, you can get data corruption known as a "race condition". WaitGroup does not prevent this.

In our next lesson, we will explore sync.Mutex, a tool designed to protect shared data by ensuring that only one goroutine can access it at a time.