Zero-Copy Buffer Passing with Pointers and Casting

The Anatomy of a Zero-Copy Buffer

In a typical communication stack, data doesn't travel alone. As it moves down from the application layer to the physical layer, each layer prepends its own header. A naive approach would be to malloc a new, larger buffer at each stage and copy the previous layer's data into it. This is exactly the expensive process we want to avoid.

The professional solution is to use a single buffer, but manage it cleverly using a descriptor. Instead of passing a simple char* pointing to the data, we pass a pointer to a struct that describes the buffer's state. This is the core idea behind the Linux kernel's famous sk_buff (socket buffer).

Let's design our own simplified version, which we'll call PacketBuffer:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// A descriptor for a packet buffer, inspired by Linux's sk_buff
typedef struct {
    unsigned char* head;   // Pointer to the start of the allocated memory
    unsigned char* data;   // Pointer to the start of the actual data
    unsigned char* tail;   // Pointer to the end of the actual data
    unsigned char* end;    // Pointer to the end of the allocated memory
} PacketBuffer;

Here is a visual representation of what these pointers track:

<-- headroom --> <--- data length ---> <--- tailroom -->
[--------------|====================|-----------------]
^              ^                    ^                 ^
head           data                 tail              end

• head & end: These pointers mark the boundaries of the entire memory chunk we allocate. They never change during the life of the buffer.
• data & tail: These are the "active" pointers. They define the window of valid data. The space between head and data is the headroom, perfect for adding headers. The space between tail and end is the tailroom, useful for appending trailers or more data.

The "zero-copy" magic happens by simply moving the data and tail pointers, without ever copying the payload.

Core Buffer Operations

Let's implement the functions to manipulate our PacketBuffer.

1. Allocation and Reserving Headroom

First, we need a function to allocate a buffer. Crucially, we'll allocate more memory than requested for the payload to create our headroom.

#define MAX_HEADER_ROOM 128 // Generous space for various protocol headers

PacketBuffer* alloc_packet(unsigned int size) {
    // Allocate memory for the descriptor and the buffer itself in one go
    unsigned char* block = malloc(sizeof(PacketBuffer) + MAX_HEADER_ROOM + size);
    if (!block) return NULL;

    PacketBuffer* pkt = (PacketBuffer*)block;
    
    // Set the boundaries of the entire allocated block
    pkt->head = block;
    pkt->end = block + sizeof(PacketBuffer) + MAX_HEADER_ROOM + size;

    // Initially, the data window is empty and located after the headroom
    pkt->data = block + sizeof(PacketBuffer) + MAX_HEADER_ROOM;
    pkt->tail = pkt->data;
    
    return pkt;
}

void free_packet(PacketBuffer* pkt) {
    // Since we allocated everything in one block, one free is sufficient
    free(pkt->head);
}

Notice we reserve MAX_HEADER_ROOM bytes. This is where headers for TCP, IP, Ethernet, etc., will live. The function alloc_packet sets the data pointer to the start of the payload area, leaving the headroom empty.

2. Adding Data (Payload)

When the application wants to write its payload, it first needs to "claim" space in the buffer. This is done by moving the tail pointer.

// Increases the data length by moving the tail pointer forward
unsigned char* packet_put(PacketBuffer* pkt, unsigned int len) {
    // Ensure there's enough tailroom
    if (pkt->tail + len > pkt->end) {
        return NULL; // Not enough space
    }
    unsigned char* old_tail = pkt->tail;
    pkt->tail += len;
    return old_tail; // Return pointer to the start of the new space
}

3. Prepending Headers: The Zero-Copy Trick

This is the most important operation. When a lower layer (e.g., the TCP layer) receives a packet from the application layer, it prepends its header by moving the data pointer backward into the headroom.

// Prepends data by moving the data pointer backward
unsigned char* packet_push(PacketBuffer* pkt, unsigned int len) {
    // Ensure there's enough headroom
    if (pkt->data - len < pkt->head + sizeof(PacketBuffer)) {
        return NULL; // Not enough space
    }
    pkt->data -= len;
    return pkt->data; // Return pointer to the start of the new header space
}

No memcpy of the payload is needed! We've just made space at the front of the buffer by adjusting a single pointer.

4. Consuming Headers

On the receiving end, the process is reversed. As the packet travels up the stack, each layer "consumes" its header by moving the data pointer forward.

// Removes data from the front by moving the data pointer forward
unsigned char* packet_pull(PacketBuffer* pkt, unsigned int len) {
    // Ensure we don't pull past the tail
    if (pkt->data + len > pkt->tail) {
        return NULL; // Not enough data to pull
    }
    pkt->data += len;
    return pkt->data; // Return pointer to the new start of data
}

From Our Model to the Real World: Linux sk_buff

This model of push, put, and pull is not just a theoretical exercise. It's a direct, simplified implementation of how the Linux kernel's networking stack works.

Putting It All Together: A Simulated Packet Flow

Now let's use our functions to simulate a packet traveling through a simplified network stack. We'll define dummy headers and use casting to place them in the buffer.

// Dummy protocol headers
typedef struct {
    unsigned short src_port;
    unsigned short dst_port;
} TCPHeader;

typedef struct {
    unsigned int src_addr;
    unsigned int dst_addr;
} IPHeader;

void print_packet_state(const PacketBuffer* pkt, const char* msg) {
    printf("%-20s: ", msg);
    printf("Data start @ %p, Length = %ld bytes\n", pkt->data, pkt->tail - pkt->data);
}

int main() {
    // --- SENDER SIDE ---

    // 1. Application Layer creates a packet
    char* payload = "Hello, Zero-Copy World!";
    unsigned int payload_len = strlen(payload) + 1;
    
    PacketBuffer* pkt = alloc_packet(payload_len);
    if (!pkt) return 1;

    print_packet_state(pkt, "Allocated");

    // 2. Application writes its data
    unsigned char* payload_location = packet_put(pkt, payload_len);
    memcpy(payload_location, payload, payload_len);
    print_packet_state(pkt, "App Layer (put)");
    printf("   -> Payload: '%s'\n", pkt->data);

    // 3. Pass to TCP Layer, which prepends its header
    unsigned char* tcp_header_location = packet_push(pkt, sizeof(TCPHeader));
    TCPHeader* tcp_hdr = (TCPHeader*)tcp_header_location;
    tcp_hdr->src_port = 12345;
    tcp_hdr->dst_port = 80;
    print_packet_state(pkt, "TCP Layer (push)");
    
    // 4. Pass to IP Layer, which prepends its header
    unsigned char* ip_header_location = packet_push(pkt, sizeof(IPHeader));
    IPHeader* ip_hdr = (IPHeader*)ip_header_location;
    ip_hdr->src_addr = 0x7F000001; // 127.0.0.1
    ip_hdr->dst_addr = 0x08080808; // 8.8.8.8
    print_packet_state(pkt, "IP Layer (push)");

    printf("\n--- PACKET TRANSMITTED ON WIRE ---\n\n");

    // --- RECEIVER SIDE ---

    // 5. IP Layer receives the packet and consumes its header
    IPHeader* received_ip_hdr = (IPHeader*)pkt->data;
    printf("Received IP Packet for 0x%08X\n", received_ip_hdr->dst_addr);
    packet_pull(pkt, sizeof(IPHeader));
    print_packet_state(pkt, "IP Layer (pull)");

    // 6. Pass to TCP Layer, which consumes its header
    TCPHeader* received_tcp_hdr = (TCPHeader*)pkt->data;
    printf("Received TCP Segment for port %d\n", received_tcp_hdr->dst_port);
    packet_pull(pkt, sizeof(TCPHeader));
    print_packet_state(pkt, "TCP Layer (pull)");

    // 7. Application receives the payload
    printf("   -> Payload: '%s'\n", pkt->data);

    free_packet(pkt);

    return 0;
}

In this example, the core payload string is never copied. We only manipulate the pkt->data pointer and cast it to the appropriate header type (TCPHeader*, IPHeader*) to write and read the protocol information. This is the essence of implementing a zero-copy mechanism with pointers and casting.

A note on casting and aliasing: Casting a char* or unsigned char* to a struct pointer to access data, as we've done, is a common idiom in C for this type of low-level programming. While strict C standards have rules against aliasing (accessing the same memory location through pointers of different types), char* is given a special exception, making this a well-defined and standard practice.

// Add this struct definition
typedef struct {
    unsigned long long dst_mac;
    unsigned long long src_mac;
} EthernetHeader;

// ... inside main, after IP Layer push ...
    
// 5. Pass to Ethernet Layer
unsigned char* eth_header_location = packet_push(pkt, sizeof(EthernetHeader));
EthernetHeader* eth_hdr = (EthernetHeader*)eth_header_location;
eth_hdr->dst_mac = 0xAABBCCDDEEFF;
eth_hdr->src_mac = 0x112233445566;
print_packet_state(pkt, "Eth Layer (push)");

// ... inside main, before IP Layer pull ...

// 5a. Ethernet Layer receives frame
EthernetHeader* received_eth_hdr = (EthernetHeader*)pkt->data;
printf("Received Ethernet Frame for MAC %llX\n", received_eth_hdr->dst_mac);
packet_pull(pkt, sizeof(EthernetHeader));
print_packet_state(pkt, "Eth Layer (pull)");

Conclusion

In this lesson, we bridged the gap between the principle of zero-copy and its practical implementation. By building a simplified buffer descriptor model, you've gained hands-on experience with the pointer manipulation techniques that underpin high-performance communication stacks in everything from Linux servers to the AUTOSAR platforms you work with.

Key Takeaways:

• A buffer descriptor struct containing pointers (head, data, tail, end) is the key to managing data without copying it.
• packet_push (moving data backward) prepends headers into pre-allocated headroom.
• packet_pull (moving data forward) consumes headers, revealing the next layer's data.
• packet_put (moving tail forward) appends data to the buffer.
• Pointer casting is essential for interpreting the raw memory of the header area as structured data (e.g., (TCPHeader*)pkt->data).

Preview of the Next Lesson

We've just seen how to manage a single packet efficiently. But a real system handles thousands of packets, often queued up for processing or transmission. How do we link these PacketBuffers together without allocating extra memory for list nodes? The answer is an intrusive linked list. In our next lesson, we will implement an intrusive linked list by embedding pointers directly within data structures, a technique that perfectly complements the buffer management system we've just built.