Designing a Simple Binary Protocol

Your Goal for Today

Learning Outcome: Design a simple binary protocol with message types for produce, consume, and acknowledge operations.

Today, we step into the role of a systems designer. We will define the rules and data structures for communication between our broker and its clients. We won't be writing the full encoding/decoding logic yet; instead, we will focus on the design of the protocol itself, specifying what messages exist and what data they contain.

This is a crucial step. A well-designed protocol is efficient and extensible, while a poorly designed one can become a performance bottleneck and a maintenance nightmare.

1. Why a Binary Protocol?

Our current echo server uses a text-based protocol. The client sends a line of text, and the server sends one back. This is simple and human-readable, but it's not efficient for high-performance systems.

For our message broker, where millions of messages might be processed per second, we need something faster and more compact. This is where binary protocols shine. Instead of sending human-readable strings like {"op": "produce", "topic": "orders"}, we send raw bytes that represent our data directly.

The benefits are significant:

• Efficiency: Parsing binary data is much faster for a computer than parsing text. There's no need to interpret characters, braces, or quotes.
• Compactness: Binary representations are almost always smaller than their text/JSON equivalents, saving network bandwidth.

Many high-performance systems use custom binary protocols. A great example to learn from is BlazingMQ, a message queuing system from Bloomberg. Their protocol documentation explains the core design principles very well.

For our broker, we will focus exclusively on the data plane—the frequent operations of sending and receiving messages. Like BlazingMQ, we'll design a set of simple, efficient binary messages for these tasks.

2. The Core Components of Our Protocol

Before defining specific messages, let's establish a few foundational concepts.

Message Types

How does the server know if a client wants to produce a message or consume one? We need to include a message type identifier in every message. This is typically a single byte or a small integer at the very beginning of the message.

Endianness

When we represent a number larger than one byte (like a 32-bit integer, uint32), we have to decide the order of the bytes.

• Big-endian: The most significant byte comes first. This is the standard for network protocols, often called "network byte order".
• Little-endian: The least significant byte comes first. This is used by most modern CPUs, like the x86 architecture.

To ensure clients and servers on different machine architectures can communicate, we must pick one and stick to it. We will use big-endian, the network standard.

3. Designing Our Message Set

Drawing inspiration from BlazingMQ's protocol, we need to define messages for the three key operations in our learning outcome: produce, consume, and acknowledge.

BlazingMQ has four main data plane messages: PutMessage (producer to broker), AckMessage (broker to producer), PushMessage (broker to consumer), and ConfirmMessage (consumer to broker). This is a solid pattern.

Let's define our own set of messages. We can start by defining the message types in Go, which will serve as our protocol specification.

// file: internal/protocol/types.go

package protocol

// MessageType defines the type of a message in our protocol.
type MessageType uint8

const (
    // ProduceRequest is sent by a client to produce a message to a topic.
    ProduceRequest MessageType = 1
    // ProduceResponse is sent by the broker to acknowledge a produce request.
    ProduceResponse MessageType = 2
    // ConsumeRequest is sent by a client to consume messages from a topic.
    ConsumeRequest MessageType = 3
    // ConsumeResponse is sent by the broker with a message for the consumer.
    ConsumeResponse MessageType = 4
    // AcknowledgeRequest is sent by the consumer to acknowledge message processing.
    AcknowledgeRequest MessageType = 5
    // AcknowledgeResponse is sent by the broker to confirm the offset commit.
    AcknowledgeResponse MessageType = 6
)

I've added responses for Acknowledge to create a symmetric request/response pattern for all operations, which simplifies client-side logic.

Now, let's define the structure for each of these messages as a Go struct. These structs will only contain fixed-size integer types (uint8, uint32, etc.), as variable-size types like string and []byte can't be directly mapped to a binary format. We'll handle them separately.

The Message Header

Every message will start with a common header. This header will tell the receiver the type of the message and its total length.

// A common header for all messages
type Header struct {
    Type         MessageType // The type of the message
    TotalLength  uint32      // Total length of the message (header + body)
}

This TotalLength field is the key to our next lesson on message framing.

Produce

• ProduceRequest: A client sends this to write a message. It needs to specify the topic and the message data.
• ProduceResponse: The broker replies with the offset where the message was stored.

// Body of a ProduceRequest message.
// The actual topic name and payload are sent after this fixed-size struct.
type ProduceRequestBody struct {
    TopicNameLength uint16
    PayloadLength   uint32
}

// Body of a ProduceResponse message.
type ProduceResponseBody struct {
    Status uint8  // 0 for success, non-zero for error
    Offset uint64 // The offset of the written message
}

Consume

• ConsumeRequest: A client sends this to read a message. It needs to specify the topic and the offset from which to start reading.
• ConsumeResponse: The broker finds a message and sends it back to the client.

// Body of a ConsumeRequest message.
// The topic name is sent after this fixed-size struct.
type ConsumeRequestBody struct {
    TopicNameLength uint16
    Offset          uint64
}

// Body of a ConsumeResponse message.
// The payload is sent after this fixed-size struct.
type ConsumeResponseBody struct {
    Status        uint8  // 0 for success, non-zero for error (e.g., no new messages)
    Offset        uint64 // The offset of this message
    PayloadLength uint32
}

Acknowledge

• AcknowledgeRequest: A consumer, after successfully processing a message, tells the broker it can commit the offset.
• AcknowledgeResponse: The broker confirms the commit.

// Body of an AcknowledgeRequest message.
// The topic name is sent after this fixed-size struct.
type AcknowledgeRequestBody struct {
    TopicNameLength uint16
    Offset          uint64 // The offset being acknowledged
}

// Body of an AcknowledgeResponse message.
type AcknowledgeResponseBody struct {
    Status uint8 // 0 for success, non-zero for error
}

4. Visualizing the Wire Format

So, what does one of these messages actually look like as a stream of bytes? Let's visualize a ProduceRequest.

The message on the wire is a sequence of the header, the fixed-size body, and then the variable-length data.

ProduceRequest for topic "orders" with payload "hello"

This structure is efficient. A receiver can first read the 5-byte header, find out the message is 22 bytes long, and then read the remaining 17 bytes from the TCP stream to get one complete, self-contained message.

The image below shows a similar breakdown for the Thrift binary protocol. You can see how it encodes fields with type information, tags, and length prefixes, just as we have designed.

As you can see, our design follows a standard pattern for binary protocols: a mix of fixed-size headers and structs, which contain length prefixes for any following variable-size data.

In the next lesson, we will use Go's encoding/binary package to implement the logic for reading and writing these structures. The articles we looked at today give you a strong foundation for what's to come. The functions binary.Read and binary.Write will be our primary tools for converting our Go structs to and from byte streams.

Conclusion

In this lesson, we put on our system designer hats and created a simple yet effective binary protocol for our message broker. This is a significant step from simply echoing text to building a system with well-defined, efficient operations.

Key Takeaways:

• Binary protocols are chosen for high-performance systems due to their compactness and parsing efficiency.
• A protocol design must specify message types, data structures, and fundamental conventions like byte order (endianness).
• Our protocol uses a common Header with a message Type and TotalLength to enable reliable message framing.
• We defined request and response messages for produce, consume, and acknowledge operations.
• Variable-length data like strings and payloads are handled by including their length in the fixed-size part of the message body.

Next Up

We have a design on paper (and in Go struct form). The next challenge is to teach our TCP server to read and write these messages. In the next lesson, we will implement length-prefixed message framing. You'll write the code to read the header, determine the total message length, and reliably read exactly one full message from the TCP stream, preparing it for decoding.