Vector Database Fundamentals

The Central Controller: The HnswIndex Struct

As a brief recap from our design session, all state associated with a database instance is managed through a single private struct. This struct is what the opaque HnswIndex* handle points to. Its definition, located in your hnsw.c implementation file, acts as the central nervous system of your library.

// In hnsw.c (The private implementation file)




#include "hnsw.h"
#include "hnsw_graph.h" // Your internal graph structures
#include "arena.h"      // Your memory allocator
#include "wal.h"        // Your write-ahead log
#include <fcntl.h>      // For file control options
#include <unistd.h>     // For ftruncate, close
#include <sys/mman.h>   // For mmap, msync, munmap

// The private definition of our index structure.
struct HnswIndex {
    int fd;                 // File descriptor for the database file
    size_t file_size;       // Current size of the file
    void* map_start;        // Start address of the memory-mapped region

    HNSWGraph* graph;       // Pointer to the graph structure (lives in the mapped memory)
    Arena data_arena;       // Arena for vector data (lives in the mapped memory)
    Arena node_arena;       // Arena for graph nodes (lives in the mapped memory)
    
    wal_t* wal;             // Pointer to the Write-Ahead Log
    // ... any other global state, e.g., configuration parameters ...
};

Every API call will receive a pointer to this struct, giving it the context needed to perform its task. Let's now implement the functions that will operate on it.

hnsw_load(): The Gateway Function

The hnsw_load function is arguably the most critical part of the API. It serves as both the constructor for new databases and the loader for existing ones. It orchestrates file I/O, memory mapping, and the data recovery process you built in the previous module.

The logic follows a clear path:

1. Allocate the Handle: First, allocate memory for the HnswIndex struct itself. This is the only part of the core data that lives on the heap; everything else will reside in the memory-mapped file.
2. Open or Create the File: Attempt to open the file at the given path. If it doesn't exist, create it.
3. Map the File: Use mmap to map the file into the process's address space.
4. Initialize or Recover:
   • If it's a new file, you must initialize the memory-mapped region. This involves setting up a header, initializing the memory arenas, and creating an empty HNSWGraph structure at the start of the mapped region. This is the "serialization" of an empty database.
   • If it's an existing file, you will deserialize the state. This means pointing the graph and arena pointers within your HnswIndex struct to the correct offsets in the mapped region. Crucially, you will then find and replay the Write-Ahead Log to recover from any potential crash and bring the database to a consistent state.
5. Return the Handle: If all steps succeed, return the pointer to the HnswIndex struct. On any failure, you must meticulously clean up any allocated resources (close file descriptors, unmap memory, free the struct) and return NULL.

This process of serialization and deserialization is fundamental to making data persistent.

When implementing this, remember the lessons on robust C programming. The code review on a kNN implementation you saw earlier highlighted the critical need for error checking.

hnsw_destroy(): The Cleanup Crew

The hnsw_destroy function is the counterpart to hnsw_load. Its purpose is to safely shut down the database instance and release all associated system resources.

1. Synchronize Data: The most important step is to call msync with the MS_SYNC flag. This flushes any "dirty" pages in the memory-mapped region to the underlying disk, guaranteeing that all insertions and modifications are durable. This directly implements the durability promise of our database.
2. Unmap Memory: Call munmap to remove the file mapping from the process's address space.
3. Close File: Close the file descriptor using close.
4. Free Handle: Finally, free the HnswIndex struct itself.

This lifecycle of load and destroy is a standard pattern for managing resources in C libraries.

hnsw_insert() and hnsw_search(): The Workhorses

These two functions are the primary interface for interacting with the data. They act as thin wrappers that delegate to the complex HNSW algorithms you've already implemented.

For hnsw_insert(HnswIndex* index, const float* vec, uint32_t id):

1. Lock (Conceptual): In a multithreaded context (which we'll explore later), this is where you'd acquire an exclusive write lock to prevent simultaneous modifications.
2. Log to WAL: Before modifying the graph, append the insertion operation (vector data and ID) to the Write-Ahead Log. This is essential for crash recovery.
3. Call Internal Insertion: Invoke your internal hnsw_graph_insert function, passing it the index->graph, the vector, the ID, and any other required parameters like M and efConstruction. This function will perform the multi-layer search and connection logic from Module 7.
4. Unlock (Conceptual): Release the write lock.
5. Return Status: Return 0 on success or a negative value on failure.

For hnsw_search(HnswIndex* index, ..., uint32_t* out_ids, float* out_dists):

1. Lock (Conceptual): Acquire a shared read lock, allowing multiple searches to happen concurrently but blocking any insertions.
2. Call Internal Search: Invoke your core hnsw_graph_search algorithm from Module 6. This function will traverse the graph from the entry point down to layer 0, using the bounded max-heap to keep track of the best candidates.
3. Copy Results: Your internal search function likely returns the results in the max-heap. You must extract the final k nearest neighbors from the heap and copy their IDs and distances into the user-provided out_ids and out_dists arrays.
4. Unlock (Conceptual): Release the read lock.
5. Return Count: Return the number of neighbors found.

A Professional Touch: Advanced Error Handling

Returning NULL or -1 is standard, but it doesn't tell the user why an operation failed. Was the file not found? Was there not enough memory? Did the data have the wrong dimensions? To build a truly robust library, you should provide more detailed error information.

A common and effective pattern in high-performance C APIs is to use a thread-local variable to store the last error message. This avoids the complexities of passing error objects around and is safe for multithreaded use. The C API for Facebook's Faiss library uses precisely this technique.

Inspired by this, you can add two things to your API:

1. A new public function in hnsw.h: const char* hnsw_get_last_error();
2. A thread-local variable in hnsw.c: _Thread_local const char* g_last_error = NULL;

Now, in your implementation of hnsw_load, if open() fails, you can do:

if (fd == -1) {
    g_last_error = "Failed to open database file";
    // ... clean up and return NULL ...
}

The user can then call hnsw_get_last_error() to get a descriptive message. This simple addition dramatically improves the usability and debuggability of your library.

Conclusion

You have now bridged the gap between your powerful internal engine and the outside world. By implementing the functions of your public API, you've created a complete, encapsulated, and durable vector database library. You've handled resource management, data persistence, and error handling in a way that is characteristic of professional, low-level systems software.

Key Takeaways:

• API Implementation: The core API functions (load, destroy, insert, search) act as controlled gateways to your library's internal logic.
• Resource Lifecycle: hnsw_load is responsible for acquiring all necessary resources (memory, file handles), while hnsw_destroy is responsible for releasing them gracefully.
• Durability via msync: The call to msync in hnsw_destroy is the final step that guarantees data written to the memory map is persisted to disk.
• Wrapping Internal Logic: hnsw_insert and hnsw_search are thin wrappers that add essential services like logging (for insert) and data marshalling around the core HNSW algorithms.
• Robust Error Reporting: Using a thread-local variable for error messages provides a clean and thread-safe way to communicate detailed error information to the user.

In the next lesson, we will move on to the final step of packaging. You will configure CMake to compile your project not as an executable, but as a dynamically linked shared library (.so on Linux), making it ready for other programs—and other programming languages—to use.