eBPF Architecture and Advantages

Introduction

In our last lesson, you mastered ftrace and trace-cmd, learning how to create a detailed, chronological log of kernel events. We saw how powerful this is for diagnosing latency and understanding complex interactions. However, we also touched on its main limitation: for high-frequency events, ftrace can generate enormous volumes of data that must be passed to user space for post-processing, which can be inefficient and overwhelm the system.

This brings us to a fundamental question: What if we could move the analysis logic itself into the kernel? What if we could write small, safe programs that run directly at the event source, perform calculations, and only send the final, summarized result back to us? This is precisely the problem that eBPF (extended Berkeley Packet Filter) solves.

In this lesson, you will describe the eBPF architecture and its advantages over traditional kernel tracing methods. We will explore how eBPF provides a safe, programmable, and highly efficient way to extend kernel functionality, making it a cornerstone of modern system observability and performance analysis.

1. The Limits of Traditional Tracing

Before diving into the eBPF architecture, it's crucial to understand the specific problems it was designed to solve. Traditional methods for kernel instrumentation each come with significant trade-offs.

Hardening eBPF for runtime security: Lessons from Datadog ...

The article 'Hardening eBPF for runtime security' from Datadog begins by evaluating the landscape of kernel monitoring tools that existed before eBPF became widespread. This provides excellent context on the limitations of those tools.

Please read the section 'What we evaluated before choosing eBPF'. As you read, categorize the main trade-offs for each method mentioned (Kernel modules, ptrace, inotify, Audit framework) in terms of safety, performance, and flexibility.

As you saw, the traditional choices were:

Kernel Modules: Extremely powerful and performant, but equally dangerous. A bug in a kernel module can easily panic the entire system. They also require maintenance to stay compatible with new kernel versions.
ftrace/perf: Much safer and built-in, but with limited programmability. They are excellent for exporting event streams, but the analysis (counting, filtering, calculating latency distributions) happens in user space.
Userspace Tools (ptrace, Auditd): Safe, but often incur significant performance penalties due to constant context switching between kernel and user space for each event.

The core challenge remained: how to achieve the performance of kernel modules with the safety of user-space tools, while also adding programmability?

The following video by Brendan Gregg, a leading expert in systems performance, perfectly demonstrates this limitation.

Give me 15 minutes and I'll change your view of Linux tracing

In 'Give me 15 minutes and I'll change your view of Linux tracing,' Brendan Gregg first showcases the power of ftrace and perf. He then pivots to a scenario where they fall short, brilliantly setting the stage for why eBPF was a necessary evolution.

Please watch from 14:24 to 18:22. From 14:24 to 15:33, pay close attention to his explanation of why calculating latency for high-frequency events like disk I/O by passing every event to user level is inefficient and 'doesn't scale'. From 15:33 to 18:22, observe how he introduces a new suite of tools based on BPF that can perform these summaries and calculations in-kernel, sending only the final result (a histogram) to the user. This is the central advantage of eBPF.

This brings us to the core value proposition of eBPF: it enables programmable, in-kernel data processing, drastically reducing the data transfer overhead and allowing for performance analysis that was previously impractical.

2. The eBPF Architecture: A Sandboxed In-Kernel VM

At its heart, eBPF is a technology that allows sandboxed programs to be executed in the Linux kernel. Think of it as a small, highly efficient virtual machine inside the kernel that can be attached to various hooks (probes, tracepoints, network sockets) to run custom code when those hooks are triggered.

Let's start with a brief overview of what eBPF is and how it evolved.

LISA21 - BPF Internals

The 'BPF Internals' talk from LISA21 gives a concise history of BPF, its evolution to eBPF, and its modern role as a generic in-kernel execution environment. It also provides a helpful table comparing eBPF to kernel and user-mode software.

Watch from 00:41 to 03:14. Focus on the transition from the original Berkeley Packet Filter to the modern, general-purpose eBPF and the key characteristics summarized in the table: performance, failure modes, and security.

The entire eBPF workflow is designed around the principles of safety and performance. Let's walk through the architectural components.

eBPF Program Lifecycle Architecture — This diagram shows the high-level data flow of an eBPF program. The process starts with a C program in user space, which is compiled into eBPF bytecode. This bytecode is then loaded into the kernel, where it undergoes strict verification before being JIT-compiled and attached to a kernel hook. Data is exchanged between the kernel and user space via eBPF maps.

Here is the step-by-step lifecycle of an eBPF program:

Development (User Space): An eBPF program is typically written in a restricted subset of C.
Compilation (User Space): A compiler like Clang/LLVM compiles the C code not into native machine code (like x86 or ARM), but into eBPF bytecode.
Loading (Kernel Space): The user-space application loads the eBPF bytecode into the kernel using the bpf() system call.
Verification (Kernel Space): This is the most critical step for safety. The kernel's Verifier statically analyzes the bytecode before it can be loaded. It performs a series of checks to prove that the program is safe to run. We'll explore this in detail shortly. If any check fails, the program is rejected.
JIT Compilation (Kernel Space): Once verified, the eBPF bytecode is translated into native machine code for the host architecture by a Just-In-Time (JIT) compiler. This makes eBPF programs run nearly as fast as natively compiled kernel code.
Attachment (Kernel Space): The verified, JIT-compiled program is attached to a specific hook point in the kernel (e.g., a tracepoint, kprobe, or network interface).
Execution (Kernel Space): When the kernel event occurs (e.g., the tracepoint is hit), the attached eBPF program executes. The program can access event data, call special helper functions, and read/write to shared data structures called maps.
Communication (Kernel-User Space): The user-space application can communicate with the running eBPF program by reading from and writing to the eBPF maps. This is how results (like counters, histograms, or filtered events) are passed back to the user.

3. The Verifier: A Mathematical Proof of Safety

The single most important component that distinguishes eBPF from kernel modules is the Verifier. It's the gatekeeper that ensures no malicious or buggy code can compromise the kernel.

The eBPF Verifier - How Linux Safely Runs User Code in Kernel Space

The article 'The eBPF Verifier - How Linux Safely Runs User Code in Kernel Space' provides an excellent and detailed explanation of how the verifier works. It breaks down the safety checks into clear categories.

Read the sections from 'The eBPF Architecture: Verified Code in a Sandbox' down to 'Failure Modes: What Happens When Things Go Wrong'. Focus on understanding the three main types of analysis the verifier performs: Control Flow Analysis, Memory Safety Verification, and Type/Context Tracking. The code examples clearly illustrate what kind of code would be accepted versus rejected.

To summarize the verifier's key guarantees:

The program must terminate: It checks for unbounded loops and ensures every code path eventually exits. This prevents an eBPF program from hanging the kernel.
No out-of-bounds memory access: The verifier tracks all memory access (to the stack, context, and maps) and ensures no pointer can read or write outside its legally permitted memory region.
No use of uninitialized variables or NULL pointers: It simulates the program's execution and ensures that every variable is initialized before use and every pointer is checked for NULL before being dereferenced.
Restricted kernel access: eBPF programs cannot call arbitrary kernel functions. They are limited to a curated set of stable helper functions (e.g., bpf_get_current_pid_tgid(), bpf_map_lookup_elem()). This provides a secure, stable API into the kernel.

Because of the verifier, a loaded eBPF program is provably safe. It cannot crash the kernel, it cannot get stuck in an infinite loop, and it cannot access arbitrary kernel memory. This gives you the power of kernel-level execution without the danger of traditional kernel modules.

4. A Deeper Look at the eBPF Workflow

Let's now watch a detailed walkthrough that follows a simple bpftrace one-liner from its text form all the way to JIT-compiled machine code running in the kernel. This will solidify your understanding of how the architectural components fit together.

LISA21 - BPF Internals

The 'BPF Internals' video provides a fantastic, in-depth tour of the entire eBPF tracing pipeline. It follows a simple example and dissects each stage of transformation and verification.

This is a longer, more technical segment, perfect for your goal of deep-level understanding. Watch from 05:37 to 28:48. Don't worry about memorizing every detail; focus on the flow: User Space to Bytecode (05:37 - 20:24): Observe how the high-level bpftrace command is parsed into an Abstract Syntax Tree (AST), then converted to LLVM Intermediate Representation (IR), and finally compiled into eBPF bytecode. This shows the role of the user-space toolchain. Verification and JIT (20:24 - 24:47): This is the core. Pay attention to how the bytecode is passed to the kernel, where the verifier runs its checks. The speaker discusses what the verifier looks for and why JIT compilation is used for both performance and security. Attachment and Output (27:39 - 28:48): See how the now-native machine code is attached to a kprobe and how data is sent back to user space via perf buffers. \nFinally, the segment from 37:39 to 38:10 provides a concise recap of this entire process.

5. Advantages of eBPF Summarized

Based on this architecture, we can now clearly articulate the advantages of eBPF over traditional methods.

Feature	eBPF	Kernel Modules	`ftrace` / `perf`
Safety	Provably safe via Verifier. Sandboxed. Cannot crash the kernel.	Unsafe. Full kernel access. A single bug can cause a panic.	Safe. Built into the kernel, actions are constrained.
Programmability	High. Custom logic can be run in-kernel (filtering, aggregation, stateful tracking).	Very High. Full C programming capabilities in kernel context.	Low. Primarily configured to stream static events. Logic is in user space.
Performance	Very High. JIT-compiled to native code. In-kernel aggregation minimizes data transfer.	Highest. Native C code running directly in the kernel.	High, but limited by data volume. Overhead increases significantly with event frequency due to kernel-user transfer.
Deployability	Easy. No kernel recompilation. Can be loaded/unloaded on a live system. CO-RE improves portability.	Difficult. Requires kernel recompilation for static modules or careful version matching for dynamic ones.	Easy. Built into the kernel.

In essence, eBPF hits the "sweet spot" by combining the safety of ftrace with the performance and programmability of kernel modules.

6. Interview Scenario

Interviewer: "We need to measure the latency distribution of a specific, high-frequency system call—say, openat—but only for requests targeting files under /etc/. The call rate is over 100,000 per second. Using ftrace to dump every enter/exit event would generate gigabytes of data per minute and overwhelm the system. How would you use eBPF to solve this efficiently?"

Click for Analysis

This scenario is tailor-made for eBPF, testing your understanding of its core advantage: in-kernel processing. A successful answer demonstrates knowledge of kprobes, maps, and the overall eBPF workflow.

Your Diagnostic Steps:

High-Level Strategy: I would write an eBPF program to perform the filtering and latency calculation entirely within the kernel, sending only the final histogram data to user space. This avoids the massive data transfer problem.
eBPF Program Design:
- Attachment Points: I'll attach eBPF programs to the entry and return points of the openat syscall. The most robust way is using raw_tracepoint/sys_enter and raw_tracepoint/sys_exit to cover all invocations, including 32-bit compatibility syscalls. Alternatively, kprobes on do_sys_openat2 or similar internal functions could be used.
- Data Structures (Maps): I'll need two eBPF maps:
  - A hash map (BPF_MAP_TYPE_HASH) to store start timestamps. It will be keyed by the thread ID (tgid). Let's call it start_ts.
  - A histogram map (BPF_MAP_TYPE_HISTOGRAM) to store the latency distribution. Let's call it latency_hist.
Execution Flow:
- On sys_enter:
  1. The eBPF program will fire. It will read the syscall arguments, specifically the pointer to the filename.
  2. It will perform an in-kernel string comparison to check if the filename starts with /etc/.
  3. If it does not match, the program exits immediately. This is the crucial filtering step.
  4. If it does match, the program gets the current timestamp using the bpf_ktime_get_ns() helper and stores it in the start_ts map, keyed by the current thread ID.
- On sys_exit:
  1. The corresponding eBPF program will fire. It looks up the start_ts map using the current thread ID to find the start time.
  2. If an entry is found, it calculates the latency: delta = bpf_ktime_get_ns() - start_timestamp.
  3. It then pushes this delta into the latency_hist map. The kernel will automatically place it in the correct logarithmic bucket.
  4. Finally, it deletes the entry from the start_ts map to clean up.
User-Space Interaction:
- A simple user-space application will periodically (e.g., every few seconds) read the latency_hist map and print the latency distribution.

Conclusion for the Interviewer:
"This eBPF approach is highly efficient because the high-frequency filtering and per-event timestamping are handled in-kernel. Only the aggregated histogram data, which is extremely small, is transferred to user space. This solves the data volume problem completely and has minimal performance impact on the target system."

Conclusion

Today you've explored the architecture of eBPF, one of the most transformative technologies in the Linux kernel in the last decade. You've seen how it moves beyond simple event reporting to enable safe, high-performance, in-kernel programmability.

Key Takeaways:

eBPF is a sandboxed virtual machine in the kernel for running user-supplied programs.
The Verifier is the cornerstone of eBPF's safety, statically analyzing bytecode to prevent crashes, infinite loops, and illegal memory access.
The JIT compiler translates verified bytecode into native machine code, ensuring high performance.
Maps are the primary mechanism for data sharing between the kernel and user space, enabling efficient in-kernel aggregation.
The key advantage of eBPF over ftrace is its ability to perform in-kernel data processing, which is crucial for analyzing high-frequency events without overwhelming the system.

Preview of the Next Lesson:

Theory is essential, but practice is where mastery is built. Now that you understand the "what" and "why" of eBPF, it's time to get your hands dirty. In the next lesson, we will move on to the practical application: Write bpftrace one-liners and use BCC tools to trace kernel behavior and analyze system performance. You will learn how to use powerful high-level tools to start leveraging eBPF for real-world analysis immediately.

Recommended resources