Writing Vector Data to Memory-Mapped Regions

The Essence of Writing to Mapped Memory

The core principle of memory-mapped I/O is that once a file is mapped into the process's address space, the operating system abstracts away the file itself. You are left with a pointer to a region of memory. If you map the file with write permissions, writing to that memory is functionally equivalent to writing to the file. The kernel's memory manager handles the complex work of propagating your changes back to the physical disk.

This is a powerful concept that fundamentally changes how you approach file I/O. Instead of fwrite or write system calls, you use memcpy or direct pointer manipulation, which is significantly more performant for many use cases as it avoids extra data copies between user-space and kernel-space buffers.

The following video from Jacob Sorber offers a clear and simple demonstration of this principle. He maps a text file and modifies its contents by simply changing characters in the resulting memory array.

For a view that aligns even more closely with the low-level C code we are writing, Andreas Kling demonstrates the same concept while building a test program for his own operating system.

Implementing the Vector Insertion Function

Now, let's translate this concept into a function for our database. First, let's define a small struct in our main header to conveniently manage the database state.

// In your main database header file (e.g., vdb.h)
typedef struct {
    int fd;                 // File descriptor
    void* mapped_region;    // Pointer to the start of the mmap'd region
    size_t file_size;       // Total size of the mapped file
    DbHeader* header;       // Pointer to the file header
} Db;

This Db struct will be our handle for an open database. When we open a file and mmap it, we'll populate this struct, including a direct pointer to the header for easy access.

Now, we can define our insertion function. It will take a pointer to our Db handle and the vector to be inserted. The logic follows four clear steps.

// In your database implementation file (e.g., vdb.c)



#include <stdio.h>
#include <string.h>
#include <stdbool.h>

// Assuming Db, DbHeader, etc. are defined in an included header
bool db_insert_vector(Db* db, float* vector_data) {
    // Step 1: Check for capacity
    if (db->header->vector_count >= db->header->vector_capacity) {
        fprintf(stderr, "Database is full. Cannot insert new vector.\n");
        return false;
    }

    // Step 2: Calculate the target address
    // The index for the new vector is simply the current count
    uint64_t index = db->header->vector_count;
    uint16_t dim = db->header->dimension;

    // The data block starts at an offset from the beginning of the mapped region
    char* base_ptr = (char*)db->mapped_region;
    float* data_block_start = (float*)(base_ptr + db->header->data_offset);

    // The destination address is at `index` offsets into the data block
    float* destination = data_block_start + (index * dim);

    // Step 3: Write the vector data using memcpy
    memcpy(destination, vector_data, dim * sizeof(float));

    // Step 4: Update the header
    db->header->vector_count++;

    return true;
}

Let's break down this implementation:

1. Check Capacity: This is a critical guard. Before writing, we must ensure there's space. We simply compare the current vector_count to the vector_capacity we defined when creating the file.
2. Calculate Target Address: This is the practical application of the pointer arithmetic we discussed in the last lesson.
   • We get the starting pointer of the entire data block by adding the data_offset from our header to the base pointer of the mapped region.
   • The new vector will be placed at the current end of the data, so its index is simply the current vector_count.
   • We calculate the final destination pointer: data_block_start + (index * dimension).
3. Write the Data: We use memcpy to copy the bytes from the source vector_data to our calculated destination. memcpy is ideal here because it's a highly optimized, standard library function for block memory copies. It's the most direct and efficient way to move our vector into place.
4. Update the Header: After successfully copying the data, we increment vector_count. This atomic-like sequence (copy data, then update count) ensures the database state remains consistent. The next insert will correctly place its data after the one we just added.

This process is remarkably efficient. There are no intermediate buffers or data transformations. We calculate a memory address and perform a single block copy directly into the memory region that the OS manages for the file. This aligns perfectly with your interest in high-performance systems.

The article on creating an append-only database provides a more complex C++ example, but the core idea is the same. It prepares a record and then writes it into a memory-mapped file.

Durability and msync (A Preview)

We have successfully written our data to the memory-mapped region. But there is a crucial subtlety: the memcpy operation modified pages in virtual memory. The operating system will eventually write these "dirty" pages back to the physical disk, but it does so at its own discretion for performance reasons. This is called a write-back cache.

What happens if your program inserts a vector and then the power goes out before the OS flushes the change to disk? The insertion will be lost.

While the kernel guarantees that changes will be flushed when a file is unmapped (e.g., your program exits cleanly via munmap), this doesn't protect against crashes. For a database, we need explicit control over durability. We need a way to tell the OS, "Write these changes to disk now."

This is precisely the purpose of the msync system call, which will be the topic of our next lesson.

Conclusion

In this lesson, we bridged the gap between file layout design and a working implementation. You now have the core function for writing data into your memory-mapped vector database.

Here are the key takeaways:

• Writing is Pointer Manipulation: Once a file is mapped with PROT_WRITE and MAP_SHARED, writing to it is as simple as calculating a pointer and using memcpy.
• The Write-Path Logic: A robust insertion function must first check for capacity, then calculate the precise offset for the new data, copy the data, and finally update the metadata (like vector_count) to reflect the change.
• Zero-Copy Efficiency: This direct-to-memory approach avoids extra buffer copies between user and kernel space, forming the foundation of our high-performance design.
• Memory vs. Disk: Changes made to a memory-mapped region are not immediately persistent on disk. The OS caches these writes for efficiency.

You have now implemented the mmap, ftruncate, and data insertion logic. The final piece of our I/O module is to gain explicit control over data durability. In the next lesson, we will explore the msync system call to ensure our written data is safely persisted to disk when we need it to be.