Structuring Data with Protobuf

From Unstructured to Structured Data

Your experience with REST APIs has likely centered around JSON, a flexible and human-readable format. However, its flexibility can also be a weakness in a large system. Without a strict schema, it's easy for a service to send data in an unexpected format, causing bugs that are hard to trace.

Protocol Buffers solves this by enforcing a rigid schema. It's a method for serializing structured data that is compact, efficient, and language-neutral.

The following video provides an excellent "crash course" that contrasts the worlds of unstructured JSON and structured Protobuf. It's a great starting point for understanding why Protobuf is so foundational in microservices.

As you saw, the key difference is the .proto file—the source of truth for your data structure.

Anatomy of a .proto File

A .proto file is a simple text file where you define your data structures, called messages. The syntax is straightforward and consists of a few key components.

Let's break down the core elements:

1. Syntax: syntax = "proto3"; is the first non-comment line and declares that you are using the proto3 syntax, which is the modern standard.
2. Message: The message keyword defines a data structure, similar to a class in Python or a struct in other languages.
3. Fields: Inside a message, you define fields, each with three parts:
   • Type: The data type, such as string, int32, or even another message type.
   • Name: The field name, typically using a snake_case convention.
   • Field Number (Tag): A unique positive integer (= 1, = 2, etc.). This number is used to identify the field in the compact binary format after serialization. These numbers are critical for schema evolution and cannot be changed once the message is in use.

Defining Messages: The Three Building Blocks

The official Google Developers documentation provides a fantastic Python-centric tutorial for defining messages. We will use its addressbook.proto example to explore the three fundamental ways to structure your data.

After reading, let's analyze the addressbook.proto example to see our three building blocks in action.

1. Scalar Types

These are the basic data types for single values. The Person message uses several:

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
  // ...
}

Here, name, id, and email are all scalar fields. Protobuf supports a wide range of scalar types like float, double, bool, and various integer sizes. For a comprehensive list, you can reference the Scalar Value Types table in the official Proto3 Language Guide (LINK).

2. Nested Messages

You can define messages inside other messages to represent complex, structured objects. The Person message contains a definition for PhoneNumber:

message Person {
  // ...
  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }
  // ...
}

Here, PhoneNumber is a nested message type. This is conceptually similar to defining an inner class in Python. It helps group related data and prevents naming collisions.

3. Repeated Fields

To represent a list or an array of items, you use the repeated keyword. The Person message can have multiple phone numbers, which is defined like this:

message Person {
  // ...
  repeated PhoneNumber phones = 4;
}

The repeated keyword in front of PhoneNumber indicates that a Person can have zero or more PhoneNumber messages. The AddressBook message at the end of the example also uses this to create a list of Person objects: repeated Person people = 1;.

This visual diagram provides a high-level summary of the entire process, from defining the schema to using it.

Practice Exercise

Now, let's apply what you've learned. Imagine you are designing the data contracts for a simplified e-commerce system.

Your task: Create a new file named order.proto and define the necessary messages.

1. Start with the syntax = "proto3"; line.
2. Define a Product message with the following fields:
   • A string for the product ID (product_id) with field number 1.
   • A string for the product name (name) with field number 2.
   • A float for the price with field number 3.
3. Define an OrderItem message that represents a product in an order. It should contain:
   • A Product field (product) with field number 1.
   • An int32 for the quantity (quantity) with field number 2.
4. Finally, define the main Order message with:
   • A string for the order ID (order_id) with field number 1.
   • A string for the customer ID (customer_id) with field number 2.
   • A repeated field of OrderItem messages (items) with field number 3.

syntax = "proto3";

// Message representing a product in the catalog.
message Product {
  string product_id = 1;
  string name = 2;
  float price = 3;
}

// Message representing a line item in an order.
// This is a nested structure, containing a Product and its quantity.
message OrderItem {
  Product product = 1;
  int32 quantity = 2;
}

// The main message representing a customer's order.
// It contains a list (repeated field) of order items.
message Order {
  string order_id = 1;
  string customer_id = 2;
  repeated OrderItem items = 3;
}

Conclusion

In this lesson, you've taken the first crucial step toward building gRPC-based microservices. You've learned how to create a .proto file to serve as a strict, language-neutral data contract between services. By understanding and using the three fundamental building blocks, you can now model complex data structures.

Key Takeaways:

• Protocol Buffers (Protobuf) provide a way to serialize structured data using a predefined schema in a .proto file.
• Messages are defined with the message keyword and contain fields.
• Each field has a type, a name, and a unique, immutable field number (tag) that is vital for binary encoding.
• The three main ways to structure data are:
  • Scalar types: For simple values like string, int32, bool.
  • Nested messages: For creating complex objects by embedding one message type within another.
  • repeated fields: For creating lists or arrays of a specific type.

Preview of the Next Lesson

You've designed your first schema, but in the real world, requirements change. What happens when you need to add a new field to your Order message, like a discount_code? If you're not careful, this could break older services that don't know about the new field.

In the next lesson, we will tackle this exact problem by exploring schema evolution. You'll learn the rules for modifying .proto files while ensuring backward compatibility, a critical skill for maintaining a healthy, evolving microservices architecture.