Binary Data Serialization

From Theory to std::ostream

At the lowest level, writing a binary file means sending a stream of raw bytes to the disk. In modern C++, the standard tool for this is std::ofstream, opened in binary mode.

#include <fstream>

std::ofstream file("mydocument.wbd", std::ios::out | std::ios::binary);
if (!file) {
    // Handle error
}

The primary method for writing raw bytes is ostream::write(const char* data, std::streamsize count). This function takes a pointer to a character buffer and writes count bytes from it directly to the file.

A very basic approach to serialization, which you might see in introductory tutorials, is to simply dump an entire C++ struct to a file.

While the technique shown in the video works for the simple case presented, it is a fragile and dangerous practice in real-world applications. As we discussed in the previous lesson, this approach fails because of:

• Padding: Compilers insert padding into structs for alignment, leading to files bloated with useless bytes that can differ between compilers or even compiler settings.
• Endianness: It assumes the reading and writing machines have the same byte order.
• Non-Trivial Types: It completely breaks for types with pointers, like std::string or std::vector, as it would write the memory address of the pointer, not the data it points to.

Therefore, we will use ostream::write as our fundamental tool, but we will apply it with surgical precision, writing our data field-by-field to maintain complete control over the output format.

A Principled BinaryWriter

To manage the serialization process cleanly, we'll create a helper class, let's call it BinaryWriter. This class will wrap an ostream and provide methods for writing the specific data types our file format requires, ensuring consistency and correctness.

This approach is common in serialization systems. The following Stack Overflow answer presents a similar concept, a BinaryContainer that writes data into an in-memory buffer. We will adapt this idea to write directly to our file stream.

Let's define our BinaryWriter. It will take a reference to an output stream and provide methods for writing our primitive types. Since our file format specifies little-endian, and we are developing on common x86/ARM platforms which are also little-endian, we can perform a direct memory write for now.

#include <fstream>
#include <cstdint>
#include <type_traits>

class BinaryWriter {
public:
    explicit BinaryWriter(std::ostream& stream) : m_stream(stream) {}

    // Write a single trivially copyable type
    template<typename T>
    void write(const T& value) {
        static_assert(std::is_trivially_copyable_v<T>, "Value must be trivially copyable.");
        m_stream.write(reinterpret_cast<const char*>(&value), sizeof(T));
    }

    // Write a block of raw bytes
    void write_bytes(const char* data, size_t size) {
        m_stream.write(data, size);
    }

private:
    std::ostream& m_stream;
};

This simple class is incredibly powerful. The template write function ensures that we can only write "plain old data" types, preventing us from accidentally trying to serialize a std::string incorrectly. When the compiler inlines this function, the resulting machine code is often just a memcpy or equivalent—achieving the "zero overhead" goal that performance-oriented developers strive for.

This concept of generating code that is as good as a handwritten memcpy is central to high-performance serialization, as discussed in the following talk.

Implementing the .wbd Writer Logic

Now we have the tools to implement the writer for our specific .wbd format. The process will be to write the file header, followed by each of our defined chunks: META, PNTS, and STRK.

A key challenge with our chunk-based format is that each chunk header (chunk_id, chunk_len) must be written before the chunk data, but we won't know the length of the data until after we've serialized it.

A clean and robust solution is to serialize each chunk's payload into a temporary in-memory buffer first. Once we have the payload in memory, we know its exact size. We can then write the chunk header to the main file, followed by the payload from our temporary buffer. std::stringstream is perfect for this.

Let's outline the top-level save function:

void saveDocument(const Whiteboard& doc, const std::string& path) {
    std::ofstream file_stream(path, std::ios::out | std::ios::binary);
    if (!file_stream) {
        // handle error
        return;
    }
    
    BinaryWriter writer(file_stream);

    // 1. Write File Header
    writer.write_bytes("WBDOC_V1", 8); // Magic number
    writer.write<uint32_t>(1);         // File version

    // 2. Write META chunk
    write_meta_chunk(writer, doc.camera);

    // 3. Write PNTS chunk
    write_paints_chunk(writer, doc.paint_palette);

    // 4. Write STRK chunk
    write_strokes_chunk(writer, doc.strokes);
}

Now, let's implement the helper functions for writing each chunk.

META Chunk Writer

This is the simplest chunk, containing fixed-size data.

void write_meta_chunk(BinaryWriter& file_writer, const Camera& camera) {
    std::stringstream chunk_buffer(std::ios::out | std::ios::in | std::ios::binary);
    BinaryWriter chunk_writer(chunk_buffer);

    // Write payload to the temporary buffer
    chunk_writer.write<double>(camera.pan_x);
    chunk_writer.write<double>(camera.pan_y);
    chunk_writer.write<double>(camera.zoom);

    // Get the payload and its size
    std::string payload = chunk_buffer.str();
    
    // Write chunk header and payload to the main file
    file_writer.write<uint32_t>('META'); // Chunk ID
    file_writer.write<uint32_t>(payload.size());
    file_writer.write_bytes(payload.data(), payload.size());
}

(Note: A 4-character literal like 'META' has an implementation-defined int value, but on little-endian systems it reliably produces the desired byte sequence M-E-T-A. This is a common and efficient idiom in low-level programming.)

PNTS and STRK Chunk Writers

These chunks contain variable-length data (lists of paints and strokes). The pattern is to first write the number of items, then loop and write each item. This is a standard serialization pattern for collections.

Here's the implementation for our STRK chunk, which is the most complex. It involves nested variable-length data (a list of strokes, each containing a list of points).

// In-memory data structures we are serializing from
struct Stroke {
    uint32_t paint_id;
    // SoA (Struct-of-Arrays) layout for points
    std::vector<float> positions_x;
    std::vector<float> positions_y;
    std::vector<float> pressures;
};

// ... inside a function like write_strokes_chunk(BinaryWriter& file_writer, const std::vector<Stroke>& strokes) ...

// Serialize payload to a temporary buffer
std::stringstream chunk_buffer(std::ios::out | std::ios::in | std::ios::binary);
BinaryWriter chunk_writer(chunk_buffer);

// Write number of strokes
chunk_writer.write<uint32_t>(strokes.size());

for (const auto& stroke : strokes) {
    chunk_writer.write<uint32_t>(stroke.paint_id);
    
    uint32_t num_points = stroke.positions_x.size();
    chunk_writer.write<uint32_t>(num_points);

    // Important: We are converting from our performant in-memory SoA layout
    // to the on-disk AoS (Array-of-Structs) point layout we designed.
    for (uint32_t i = 0; i < num_points; ++i) {
        chunk_writer.write<float>(stroke.positions_x[i]);
        chunk_writer.write<float>(stroke.positions_y[i]);
        chunk_writer.write<float>(stroke.pressures[i]);
    }
}

// Now write the completed chunk payload to the file stream...
// (Code to write chunk header and payload is the same as in write_meta_chunk)

This code precisely implements the STRK chunk format we designed. It correctly handles the nested structure and demonstrates a key concept: the in-memory data layout (SoA for performance) can be different from the on-disk layout (AoS for easier parsing), with the serializer acting as the bridge between them.

Conclusion

In this lesson, you have translated a theoretical file format design into a robust and performant C++ implementation. By building a dedicated BinaryWriter class, you've created a clean, safe, and extensible system for serializing your application's state.

Key Takeaways:

• Controlled Writing: Never dump raw C++ structs to a file. Use a writer class to serialize data field-by-field, giving you full control over the binary layout.
• The size -> data Pattern: This is the fundamental pattern for serializing variable-length collections like arrays, vectors, and strings.
• Buffer-First for Chunks: For chunk-based formats, serializing a chunk's payload to an in-memory buffer first is a clean way to determine its length before writing the chunk header.
• Zero-Overhead Goal: Simple, inlined functions that perform direct memory writes (like our BinaryWriter::write) are the key to achieving performance that rivals handwritten serialization code.

You now have a complete writer capable of saving your whiteboard documents. In the next lesson, we will build its counterpart: you will implement a reader to deserialize a whiteboard document from the binary format, correctly reconstructing the application state. This will involve reading the very files we just learned how to create, closing the loop on data persistence.