Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 55

Kernel Crash Dump Analysis with 'crash' Utility

Recommended resources

Introduction

Welcome back! In our previous lesson, we successfully configured kdump to act as a flight recorder for the kernel, capturing a complete memory snapshot—a vmcore file—at the moment of a panic. This is a crucial first step in post-mortem debugging. However, a multi-gigabyte raw memory file isn't useful on its own. The real detective work begins now.

This lesson directly follows our last session. We will take the vmcore file we learned to create and load it into the crash utility. You will learn to analyze a kernel crash dump using 'crash' to perform a post-mortem root cause analysis. We'll start with the basics of interpreting the initial crash report and then move through a systematic process of using core commands to pinpoint the exact line of code that caused the failure.

The crash utility is a powerful tool that combines the symbol-parsing capabilities of GDB with an awareness of kernel data structures, making it the standard for Linux crash dump analysis.

1. Launching crash and The Initial Report

The crash utility needs two things to function:

  1. The uncompressed kernel image (vmlinux) with debugging symbols. This acts as a map, telling crash where functions and data structures are located.
  2. The memory dump file (vmcore). This is the raw data—the state of memory at the time of the crash.

You launch it from your host machine's terminal like this:

crash /path/to/debug/kernel/vmlinux /path/to/dump/vmcore

Note: Ensure the vmlinux file is the exact one that was running when the kernel panicked.

Upon loading, crash provides an initial summary report. This is your first set of clues.

Analyzing Linux kernel crash dumps with crash

The article 'Analyzing Linux kernel crash dumps with crash' by Dedoimedo provides an excellent, detailed walkthrough of the crash utility. Let's start by understanding the initial output it presents.

Read the section 'Analyzing the crash report - First steps'. Focus on understanding the meaning of each field in the initial report, from KERNEL and DUMPFILE down to STATE. Pay close attention to the PANIC string, which often contains the kernel's own summary of what went wrong.

Let's break down the key fields you just read about, using a typical example output.

Initial 'crash' Utility Output for Kernel Crash Dump Analysis
This is the initial report from the `crash` utility. It provides a high-level summary of the system's state at the time of the panic, including the kernel version, timestamp, number of tasks, and most importantly, the panic message and the process that was running.

From this report, you can immediately determine:

  • KERNEL & DUMPFILE: The files you are analyzing.
  • DATE: When the crash occurred.
  • PANIC: The reason for the panic. This could be a manually triggered SysRq, a "Kernel panic - not syncing" message, or an "Oops".
  • PID, COMMAND, TASK: The Process ID, name, and task struct address of the process that was active on the crashing CPU.
  • CPU: Which CPU core the crash occurred on.
  • STATE: The state of the active task (e.g., TASK_RUNNING).

Understanding "Oops" Codes

Sometimes, the PANIC string will contain a message like Oops: 0002. This isn't random; it's a bitmask that describes a page fault. Understanding this code gives you a head start.

Analyzing Linux kernel crash dumps with crash

The Dedoimedo article also deciphers these Oops codes. This is a great piece of 'atomic' knowledge that is very useful in practice.

Read the subsections 'Another example, from the White Paper' and 'Kernel Page Error'. Focus on how the decimal code (e.g., 0002) is translated into a binary bitfield to describe the fault.

To summarize the Oops code 0002:

  • Decimal 2 is binary 0010.
  • We read the bits from right to left (bit 0, bit 1, etc.).
  • Bit 0 (PF_PROT): 0 -> Access to a page that is not present.
  • Bit 1 (PF_WRITE): 1 -> The access was a write operation.
  • Bit 2 (PF_USER): 0 -> The fault occurred in Kernel Mode.
  • Bit 3 (PF_RSVD): 0 -> Not a reserved bit violation.
  • Bit 4 (PF_INSTR): 0 -> Not an instruction fetch.

So, Oops: 0002 immediately tells you the kernel tried to write to a memory address that wasn't mapped, while in kernel mode. A classic symptom of a NULL pointer dereference or a corrupted pointer.

2. The Core Analysis Workflow

With the initial context set, you can begin a systematic investigation using a few key commands.

Backtrace (bt): The Story of the Crash

The bt command is your most important tool. It shows the kernel stack for the active task, revealing the sequence of function calls that led to the crash.

Kernel Crash Utility Backtrace Example
The backtrace (`bt`) command shows the call stack leading to the exception. You read it from the bottom up to see the function call history. The `exception RIP` line pinpoints the exact function and instruction that failed.

Let's dissect this output, using the excellent explanation from the Dedoimedo guide.

Analyzing Linux kernel crash dumps with crash

Now let's dive into the most important command, bt.

Read the 'Getting hot' section, focusing on 'Backtrace', 'Instruction pointer', and 'Code Segment (CS) register'. This will explain how to interpret the call trace, the RIP (Instruction Pointer), and the CS register to determine the privilege level.

Key takeaways from the backtrace:

  1. Read from bottom to top: In the image above, cpu_idle called default_idle, which eventually led to the crash.
  2. Find the exception: The line [exception RIP: default_idle+61] tells you the crash happened in the default_idle function, at an offset of 61 bytes from the function's start. RIP stands for Instruction Pointer.
  3. Check the privilege level: The CS: 0010 value indicates the Code Segment. As the article explains, the last two bits determine the privilege level. Since 0010 is binary, this corresponds to CPL 0 (Kernel Mode). A crash in User Mode (CPL 3, e.g., CS: 0033) would point towards a hardware issue or a bug in how a system call handles invalid user input.

Kernel Log (log): Events Preceding the Crash

The bt command shows you the moment of the crash. The log command shows you the moments leading up to it. It dumps the kernel's ring buffer (dmesg), which might contain warnings or error messages that are crucial clues.

crash> log

Always run log early in your analysis. You might find driver error messages, memory warnings, or other events that provide context for the final panic.

Process Status (ps): Who Was Running?

The ps command shows you the state of all processes and threads at the time of the crash. This is useful for understanding the overall system workload and confirming which process was the "offending task."

crash> ps

The active task on each CPU is marked with a >. This should match the PID/COMMAND from the initial report.

3. Practical Scenario: Root-Causing a NULL Pointer Crash

Theory is useful, but let's apply it. This is the kind of task you'd get in a high-level interview or a real-world debugging session.

Scenario: An embedded device panics immediately after you load a new, experimental GPIO driver module (null-pointer.ko). You have captured the vmcore. Let's find the bug.

We'll follow the exact workflow laid out in the Dedoimedo article, which provides a perfect, self-contained example.

Analyzing Linux kernel crash dumps with crash

This is the core of our lesson. We will walk through a complete analysis of a crash caused by a buggy kernel module. This combines everything we've learned into a practical workflow.

Read the entire 'Difficult example' section. Follow the steps of the analysis closely: Note the PANIC: "Oops: 0002" message and what it implies. See how the log command is used to find the RIP, which points to a function within our buggy module (null_pointer:init_module+...). Understand the need to disassemble the module's code (.ko file) to see the assembly instructions. Follow the final step where the register dump from the log (RAX: 0000000000000000) is correlated with the assembly instruction (movb $0x1,(%rax)) to prove a write to address 0.

Let's summarize the analytical process you just read about:

  1. Launch crash: crash vmlinux vmcore
  2. Initial Analysis: The report shows PANIC: "Oops: 0002". We know this means a write to a non-present page in kernel mode.
  3. Check the Log: The log command shows a trace, and the RIP points directly to our module: RIP: ... null_pointer:init_module+0x19/0x22. This confirms our module is the source of the crash. The log also contains the register dump, showing RAX is all zeros.
  4. Load Module Symbols: For crash to understand the functions inside a kernel module, you need to load its symbols. The Dedoimedo article uses objdump externally, but a more integrated way is to use the mod command inside crash. 
    crash> mod -S /path/to/module/null-pointer.ko
  5. Disassemble the Code: Now that symbols are loaded, you can disassemble the problematic function directly within crash. 
    crash> dis init_module
...
0xffffffffc06c8019 <init_module+25>: movb   $0x1,(%rax)
...
  6. Connect the Dots (The "Aha!" Moment):
    • The log showed us that at the time of the crash, the RAX register contained 0x0000000000000000.
    • The disassembly shows that the instruction at the crash address (init_module+0x19, which is init_module+25 in decimal) was movb $0x1,(%rax).
    • This instruction attempts to move the value 1 into the memory location pointed to by the RAX register.
    • Conclusion: The kernel tried to write the value 1 to memory address 0. This is a classic NULL pointer dereference, which caused the page fault and the kernel panic. The bug is in the C code: *p = 1; where p was a NULL pointer.

4. Going Deeper: Finding Local Variable Values

Sometimes, you need to know the value of a local variable within a function in the call stack, but crash (like GDB) can sometimes struggle to retrieve it directly. For this, you can resort to a more manual, assembly-level analysis—a technique that demonstrates deep system knowledge.

A crash course on debugging kernel crashes using the crash utility

Let's look at a more advanced technique from a real-world debugging session described by Wander Lairson Costa. This shows how to manually find local variable values by inspecting the stack frame and understanding the CPU's calling convention.

Read the final, and most detailed, part of this article. You don't need to follow every single address. Focus on the process: He starts by looking at the function's assembly code with dis. He identifies where the function parameters (passed in registers like %rdi, %rsi) and local variables are saved on the stack, relative to the base pointer (%rbp). He uses bt -f to get a full dump of the raw stack frame contents. He manually calculates the memory address of the variable on the stack (rbp - offset) and reads its value from the bt -f output.

This manual technique of correlating the calling convention, function prologue, and raw stack dump is an advanced skill. Being able to describe this process shows an exceptional level of understanding of how the system works at a low level.

Conclusion

You've now completed the full post-mortem debugging cycle: configuring kdump, capturing a vmcore, and performing a root cause analysis with crash. This is one of the most fundamental and powerful skills for a kernel developer.

Key Takeaways:

  • Systematic Approach: A successful analysis follows a clear workflow: get context from the initial report and log, get the call history from bt, and then dive into the specific code and memory state with dis and other commands.
  • Backtrace is King: The bt command is your starting point for almost every investigation.
  • Correlate Information: The solution is rarely in one place. You must connect clues from the panic string, the log, the backtrace, and the register state to form a complete picture.
  • Know Your Tools: Mastering commands like bt, log, ps, mod, and dis is essential.

Interview Scenario:
Interviewer: "A vmcore analysis shows a kernel panic inside a complex driver function. The bt command points to a specific line, but to understand why the crash happened, you need to know the value of a local variable named device_state. For some reason, print device_state isn't working. How could you determine its value?"

Your response should outline the advanced technique we just covered: explain that you would use dis on the function to find where device_state is stored on the stack relative to the base pointer (%rbp), then use bt -f to dump the raw stack contents and manually read the value from the calculated address.

Preview of the Next Lesson:

crash is the definitive tool for post-mortem analysis of a host or bare-metal kernel. But what if you're developing for virtualized systems? How do you debug a guest kernel while it's running? In our next lesson, "Debug guest kernel code using QEMU's GDB stub to step through virtualized kernel execution," we will explore the world of live debugging in a virtualized environment.