Go Structs for Message Broker Core Components

Introduction to the Lesson

This lesson marks the beginning of a new module: Designing the Broker's Storage Engine. We will shift our focus from the network to the heart of the broker—the logic that manages and stores the messages.

Your Goal for Today:
Your learning outcome is to define the core Go structs for topics, partitions, and messages in the message broker.

This is the first and most crucial step in building the storage layer. By translating the abstract concepts of a message broker into concrete Go data structures, we'll create the logical skeleton upon which all future storage and retrieval logic will be built.

1. The Core Concepts: Topics, Partitions, and Messages

Before we write any code, let's establish a clear mental model. Message brokers like Kafka organize data into a simple but powerful hierarchy:

• Message: The smallest unit of data. It's the information a producer sends and a consumer reads. Each message has content (its payload) and a unique position within its log.
• Partition: A sequence of messages, structured as an ordered, immutable, append-only log. Each message within a partition is assigned a sequential ID number called its offset, which uniquely identifies it within that partition.
• Topic: A named category or feed to which messages are published. A topic is composed of one or more partitions. Splitting a topic into multiple partitions is the key to achieving scalability and parallelism, as different consumers can read from different partitions simultaneously.

This structure allows a broker to handle high-throughput data streams efficiently. The image below illustrates how messages for a single topic are distributed across several partitions within a broker.

Within each partition, the messages form an ordered log, as shown here.

Our task is to model this Topic -> Partition -> Message hierarchy using Go's struct type.

2. From Concepts to Go Structs

Let's translate these ideas into code. Since you have extensive experience as a software engineer, you'll recognize this as a classic domain modeling exercise. We'll start with the smallest unit, the Message, and work our way up.

For this part of the project, let's create a new package to house our broker's internal logic.

1. Create a new directory: internal/broker.
2. Inside that directory, create a new file: broker.go.

Now, let's define our structs in this new file.

The Message Struct

A message is simple. For now, it needs to contain the data itself and its position in the partition's log.

// internal/broker/broker.go
package broker

// Message is the fundamental data unit, representing a single log entry.
type Message struct {
    Offset uint64
    Data   []byte
}

The Partition Struct

A partition is essentially a log of messages. An in-memory slice is a good starting point for this log ([]*Message). Critically, multiple goroutines (one for each connected client) will be trying to write to and read from this partition simultaneously. To prevent data corruption, we must protect access to it. The sync.RWMutex is a perfect tool for this, as it allows multiple readers to access the data concurrently but ensures exclusive access for writers.

// internal/broker/broker.go
package broker

import "sync"

// ... Message struct from above

// Partition represents an ordered, append-only log of messages.
// Access to a partition's messages must be synchronized.
type Partition struct {
    ID       uint64
    mu       sync.RWMutex
    messages []*Message
}

The Topic Struct

Finally, a topic manages a collection of partitions. A map is a natural fit for this, allowing us to look up a partition by its ID. Just like the partition itself, this map is shared state that will be accessed concurrently (e.g., when creating new partitions or when a producer needs to find a specific partition). Therefore, it also requires a sync.RWMutex.

// internal/broker/broker.go
package broker

// ... imports and other structs

// Topic groups related partitions.
// Access to the topic's partition map must be synchronized.
type Topic struct {
    Name       string
    mu         sync.RWMutex
    partitions map[uint64]*Partition
}

We now have the basic data structures that represent the core components of our broker.

To see another example of this modeling, let's look at a similar project.

3. Initializing the Structs

Defining structs is one thing; creating instances of them is the next. Let's add simple constructor functions for our Topic and Partition to ensure they are initialized correctly. This is good practice and will be essential as our logic grows more complex.

Add these functions to your internal/broker/broker.go file.

// internal/broker/broker.go

// ... (after the struct definitions)

// NewPartition creates and returns a new Partition.
func NewPartition(id uint64) *Partition {
    return &Partition{
        ID:       id,
        messages: make([]*Message, 0),
        // The zero value for a sync.RWMutex is a valid, unlocked mutex.
    }
}

// NewTopic creates and returns a new Topic.
func NewTopic(name string) *Topic {
    return &Topic{
        Name:       name,
        partitions: make(map[uint64]*Partition),
    }
}

With these definitions in internal/broker/broker.go, you have successfully laid the conceptual and structural foundation for the broker's storage engine. For now, this is all just in-memory. In later lessons, we will build on this foundation to create a swappable storage layer that can write to a file or a database like PostgreSQL.

This approach—defining simple in-memory data structures first—allows us to focus on the core logic before tackling the complexities of persistent storage and data serialization, a topic explored in the "Distributed Services with Go" project (resource LINK). That project uses Protocol Buffers to define messages and builds a sophisticated on-disk log with segments and indexes, representing a more production-grade implementation that we will be working towards.

Conclusion

In this lesson, you designed and implemented the fundamental data models for our message broker. This is a critical step that moves us from handling network packets to managing application-level data.

Key Takeaways:

• A scalable message broker is typically built on a hierarchy of Topics, Partitions, and Messages.
• Go structs are the ideal tool for modeling these domain concepts in code.
• Any state that will be accessed concurrently by multiple goroutines, such as a topic's list of partitions or a partition's log of messages, must be protected with a synchronization primitive like sync.RWMutex.

Next Up

You've defined the what—the data structures that hold our state. In the next lesson, you'll start working on the how. We will implement the logic that uses these structs to manage the broker's state. You will put the sync.RWMutex you just added into practice to protect concurrent access to a central map of topics, ensuring that our broker can safely handle requests to create and retrieve topics from multiple clients at the same time.