Debugging the Linux Kernel with KGDB/KDB

1. Understanding KGDB and KDB

Before we configure anything, it's crucial to understand what these tools are and how they relate to each other. KGDB provides the back-end for source-level debugging with GDB, while KDB offers a more direct, command-line interface for quick analysis.

In essence, you get the best of both worlds. You can use KDB for a quick look at the system state, and then seamlessly switch to KGDB for a full-powered GDB session when you need to dig deeper.

2. Configuring the Target Environment

Setting up for kernel debugging is the most critical and detailed part of the process. It involves two main steps: building a kernel with the correct features enabled and configuring the system to make the debugger available.

The standard setup requires two machines:

• Target: The machine (physical or virtual) running the custom-built, debug-enabled kernel.
• Host: Your development machine, where you run the GDB client.

The connection is typically made over a serial port, which can be a physical port or a virtual one provided by a hypervisor like QEMU.

2.1. Kernel Build Configuration

You must compile your kernel from source with specific options enabled. These options instrument the kernel and include the necessary debugging symbols and infrastructure.

Here is a checklist of the essential kernel configurations you'll need. You can find these options in your kernel's .config file or via make menuconfig under "Kernel hacking".

Essential Configuration Checklist:

• CONFIG_KGDB=y: Enables the core kernel debugger infrastructure.
• CONFIG_KGDB_SERIAL_CONSOLE=y: Enables the driver to use a serial port for debugging. This is the most common I/O mechanism.
• CONFIG_KGDB_KDB=y: Includes the KDB shell front-end. This allows you to use both KDB and KGDB.
• CONFIG_DEBUG_INFO=y: Crucial. This tells GCC to include DWARF debugging symbols in the vmlinux file, which GDB needs for source-level debugging.
• CONFIG_FRAME_POINTER=y: Helps GDB produce more reliable stack backtraces.
• CONFIG_MAGIC_SYSRQ=y: Enables the Magic System Request key, which is the primary way to manually enter the debugger on a running system.

Important Considerations for Debugging:

• Disable Kernel Address Space Layout Randomization (KASLR): KASLR moves the kernel's location in memory on every boot, which confuses GDB. You can disable it with CONFIG_RANDOMIZE_BASE=n or, more flexibly, by adding the nokaslr parameter to the kernel's boot command line.
• Disable Watchdogs: When the kernel is stopped in the debugger, it cannot service watchdog timers. This will cause the system to reboot unexpectedly. Disable any hardware or software watchdogs (CONFIG_WATCHDOG=n and check your bootloader settings).
• Disable Read-Only Protections: For GDB to insert software breakpoints, it needs to write to the kernel's code section. Features like CONFIG_STRICT_KERNEL_RWX prevent this. It's often easiest to disable this during a debug session.

2.2. Hardware Setup and Boot Arguments

With your custom kernel compiled, you need to boot it on the target and tell it how to communicate with the debugger.

The two most important boot arguments are:

1. kgdboc=<device>,[baud]: This tells the "KGDB over console" driver which serial device to use and at what speed. For example: kgdboc=ttyS0,115200.
2. kgdbwait: This forces the kernel to stop early in the boot process and wait for a GDB client to connect. This is essential for debugging issues that happen during startup, such as in driver probe functions.

A typical debug-ready kernel command line in your bootloader might look like this:
console=ttyS0,115200 kgdboc=ttyS0,115200 kgdbwait nokaslr

3. Using the Kernel Debugger

With the environment configured, you are ready to stop the kernel and inspect it.

3.1. Entering the Debugger and Using KDB

The most common way to manually break into the debugger on a running system is with a Magic SysRq command.

1. From a shell on the target machine, run:

   echo g > /proc/sysrq-trigger
   

2. The kernel will halt, and you will see a kdb> prompt on your serial console.

Now you are in the KDB shell. It's a simple but effective tool for a first look.

Here are some of the most useful KDB commands:

• help: Lists all available commands.
• ps: Shows the process list.
• dmesg: Dumps the kernel log buffer.
• bt: Prints a backtrace of the currently executing task.
• btc: (On multi-core systems) Prints a backtrace for the task running on each CPU.
• go: Resumes kernel execution.
• kgdb: Switches from the KDB shell to KGDB mode, preparing to accept a GDB connection.

3.2. Source-Level Debugging with KGDB and GDB

Now for the main event: connecting GDB for full source-level debugging.

The Workflow:

1. Stop the kernel: Enter the debugger via SysRq, a panic, or kgdbwait. If you're in the KDB shell, you can optionally type kgdb to explicitly enter KGDB mode.
2. Launch GDB on the host: You must use a GDB built for your target's architecture (e.g., aarch64-linux-gnu-gdb) and point it to the vmlinux file that contains the debug symbols.

   
   
   
   # On your host machine
   aarch64-linux-gnu-gdb /path/to/your/kernel/source/vmlinux
   

3. Connect to the target: Use the target remote command, pointing to the serial port device on your host that is connected to the target.

   
   
   
   # Inside GDB
   (gdb) set remotebaud 115200
   (gdb) target remote /dev/ttyUSB0
   

You are now connected! GDB will show you the exact line of C code where the kernel is stopped.

3.3. Advanced Inspection: Debugging Other Tasks

When you first connect, you are debugging the task that was running on the CPU that entered the debugger. But what if the problem lies in another process or even on another CPU? KGDB represents all kernel tasks (processes/threads) and CPUs as GDB "threads".

You can use standard GDB thread commands to investigate the entire system.

This is an incredibly powerful technique that allows you to move around a completely frozen system and inspect the state of any task you wish.

A Note on Modern Front-Ends

While powerful, the GDB command line can feel archaic. Modern IDEs like Visual Studio Code can act as a graphical front-end for GDB, providing a much more intuitive experience with clickable breakpoints, variable watch windows, and call stack navigation.

4. Interview Scenario

Interviewer: "A kernel panic is occurring intermittently on one of our embedded devices during the initialization of a specific network driver. We have serial console access. A simple printk in the driver's probe function isn't enough to pinpoint the issue. Describe, step-by-step, how you would use a kernel debugger to find the root cause of this panic, assuming it happens after the kernel has started booting but before you can log in."

Conclusion

You have now learned one of the most powerful techniques in the kernel developer's arsenal. While logging provides the breadcrumbs, interactive debugging with KGDB and GDB lets you become a detective, freezing the scene of the crime and examining every piece of evidence.

Key Takeaways:

• Configuration is Key: A successful debugging session depends on a correctly configured environment, including the right kernel CONFIG options and boot arguments (kgdboc, kgdbwait, nokaslr).
• KDB is for Quick Triage: Use the KDB shell for a fast, initial assessment of the system state with commands like bt, ps, and dmesg.
• KGDB/GDB is for Deep Analysis: For true source-level debugging, connect a GDB client to the KGDB stub. This allows you to set breakpoints, step through code, and inspect variables and memory as you would in an application.
• The Workflow: The standard process is to stop the kernel (via kgdbwait or SysRq), connect GDB to the vmlinux file, set a breakpoint in the area of interest, and then step through the code to find the fault.

Preview of the Next Lesson:

Debugging a live kernel is powerful, but you can't always keep a production system halted. Often, a system will crash and reboot immediately. In the next lesson, "Configure the kdump service to capture a kernel crash dump upon a panic," we will learn how to set up a system to automatically save the entire state of main memory when a crash occurs. This allows us to perform a post-mortem analysis, applying many of the same debugging skills to a crash dump file long after the event.