Benchmarking Producer Throughput with testing.B

1. The Go Benchmarking Toolkit

Just as Go provides the testing package for unit tests, it offers a powerful, integrated framework for writing performance benchmarks. These benchmarks live in the same *_test.go files and use a similar naming convention.

The fundamental components are:

• Benchmark Functions: Named BenchmarkXxx, they accept a single argument: b *testing.B.
• The Benchmark Runner: The go test command, when run with the -bench flag, executes these functions.
• The b.N Loop: The core of a benchmark is a loop that runs b.N times. The Go tool automatically adjusts N until the benchmark has run long enough to gather statistically stable results.

Let's start with a video that introduces these core concepts in a very practical way.

To solidify these fundamentals, let's turn to a comprehensive guide that formalizes what you just saw and introduces a few more crucial concepts, like controlling the timer.

A key takeaway here is the separation of setup from the work being measured. Any expensive initialization (like creating data structures, reading files, or setting up servers) should happen before the for loop and, importantly, before b.ResetTimer().

2. Measuring Performance Under Concurrent Load

The for i := 0; i < b.N; i++ loop is perfect for measuring a single, sequential operation. However, our goal is to measure throughput under concurrent load. We want to simulate many producers sending messages at the same time, which is a much more realistic scenario for a message broker.

A simple loop won't work because it executes the operations one after another. For true concurrency, we need to use a different tool: b.RunParallel.

b.RunParallel is a powerful function that:

1. Creates multiple goroutines (by default, GOMAXPROCS of them).
2. Distributes the b.N iterations among these goroutines.
3. Waits for all of them to complete.

Inside b.RunParallel, you provide a function that contains another loop: for pb.Next() { ... }. Each goroutine runs this loop until its share of the b.N iterations is complete.

This is the standard and correct way to benchmark the performance of concurrent code in Go.

The article highlights a crucial pattern: shared state (like the sync.Map in the example) is defined outside the b.RunParallel call, while goroutine-local state is defined inside it, but before the pb.Next() loop. This is exactly the model we will follow.

3. Building the Broker's Throughput Benchmark

Now, let's apply these concepts to build a benchmark for our producer. We will create a new file, broker_benchmark_test.go, inside your broker package.

The plan is as follows:

1. Global Setup: In the BenchmarkProducerThroughput function, we'll start a single instance of our broker. This broker will be shared by all concurrent clients.
2. Reset Timer: After the broker is running and we've done any other setup (like creating a topic), we'll call b.ResetTimer() once.
3. Run in Parallel: We'll call b.RunParallel.
4. Goroutine-Local Setup: Inside the function passed to b.RunParallel, each goroutine will create its own independent client connection to the broker. This is critical for simulating multiple, distinct producers.
5. The Measured Operation: Inside the for pb.Next() { ... } loop, each goroutine will call its client's Produce method to send a message. This is the core operation whose performance we are measuring.

Here is what the implementation will look like. You can add this to a new broker_benchmark_test.go file.

package broker

import (
	"context"
	"testing"
	"time"

	// You will need to import your actual client and storage packages
	"your-project-path/pkg/client" 
	"your-project-path/pkg/storage"
)

// setupTestBroker is a helper to create a broker instance for testing/benchmarking.
// It's good practice to keep setup logic separate.
// This should create a broker with an in-memory log for speed, if you have one.
func setupTestBroker(t testing.TB) *Broker {
	// For this benchmark, you might want a configuration that is fast,
	// e.g., using in-memory storage or disabling complex features not under test.
	// For now, let's assume a simple NewBroker() call works.
	// In a real project, you would pass a test-specific configuration.
	broker, err := NewBroker(Config{ListenAddr: ":0"}) // :0 lets the OS pick a free port
	if err != nil {
		t.Fatalf("failed to create test broker: %v", err)
	}
	return broker
}

func BenchmarkProducerThroughput(b *testing.B) {
	// --- 1. Global Setup ---
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	broker := setupTestBroker(b)
	go func() {
		// Start the broker but ignore the error on shutdown.
		_ = broker.Start(ctx)
	}()

	// Wait a moment for the broker to start listening.
	// A more robust solution would be to use a channel to signal readiness.
	time.Sleep(100 * time.Millisecond)

	topic := "benchmark-throughput-topic"
	_ = broker.createTopic(ctx, topic) // Assuming internal method access for simplicity

	// Prepare a sample message that we'll reuse.
	payload := []byte("a moderately sized message for benchmarking purposes")
	
	// --- 2. Reset Timer & Report Allocations ---
	b.ResetTimer()
	b.ReportAllocs()

	// --- 3. Run in Parallel ---
	b.RunParallel(func(pb *testing.PB) {
		// --- 4. Goroutine-Local Setup ---
		// Each goroutine gets its own client to simulate concurrent producers.
		c, err := client.New(broker.Addr())
		if err != nil {
			// Using b.Fatal in a parallel test can be tricky.
			// It's often better to signal an error and check it later.
			b.Errorf("failed to create client: %v", err)
			return
		}
		defer c.Close()

		// --- 5. The Measured Operation ---
		// pb.Next() returns true until its share of b.N iterations are done.
		for pb.Next() {
			err := c.Produce(ctx, topic, payload)
			if err != nil {
				// Don't use b.Fatal here as it stops everything.
				// In a real test, you might use an atomic counter to track errors.
			}
		}
	})
}

To run your new benchmark, navigate to your project's broker directory in the terminal and execute:

go test -bench=ProducerThroughput -benchmem

• -bench=ProducerThroughput tells Go to run only benchmarks whose names match the regular expression ProducerThroughput.
• -benchmem is a crucial flag that instructs the benchmark runner to report memory allocation statistics (B/op and allocs/op).

The output will look something like this:

goos: darwin
goarch: arm64
pkg: your-project-path/broker
BenchmarkProducerThroughput-10    137666    8556 ns/op    152 B/op    4 allocs/op
PASS
ok      your-project-path/broker    2.134s

• BenchmarkProducerThroughput-10: Ran on 10 CPU cores.
• 137666: The final value of b.N (total operations performed).
• 8556 ns/op: Each "produce" operation took, on average, 8556 nanoseconds. This is our throughput metric.
• 152 B/op: Each operation allocated 152 bytes of memory.
• 4 allocs/op: Each operation caused 4 memory allocations.

Lower is better for all these numbers. To convert ns/op to operations per second (throughput):
Throughput = 1,000,000,000 / (ns/op)
In this example: 1,000,000,000 / 8556 ≈ 116,898 ops/sec.

This type of data is invaluable. It gives you a baseline to compare against as you make changes. For example, you can now run this benchmark with different message sizes or after a code change to see the performance impact.

Conclusion

Congratulations! You have written your first concurrent benchmark and have a quantitative measure of your broker's producer throughput. This is a fundamental skill in systems programming and is essential for building high-performance software.

Key Takeaways:

• Go's built-in testing package provides a first-class framework for performance benchmarking.
• Simple, sequential benchmarks use a for i := 0; i < b.N; i++ loop.
• b.RunParallel is the standard tool for measuring performance under concurrent load, simulating real-world usage patterns.
• Careful benchmark design involves separating one-time setup from the code being measured (b.ResetTimer) and distinguishing between shared state and goroutine-local state.
• The output (ns/op, B/op, allocs/op) provides a clear, quantitative basis for performance analysis and optimization.

Next Up

Throughput tells us how many operations our system can handle per second, but it doesn't tell the whole story. A system can have high throughput but also unacceptably high latency. In the next lesson, we will build on our new benchmarking skills to measure end-to-end message latency: the total time it takes for a message to go from producer to consumer.