Configuring Kdump for Kernel Crash Dumps

1. The Kdump Mechanism: Two Kernels are Better Than One

When a kernel panics, it's in an unstable state. It cannot be trusted to perform complex operations like writing its own memory image to a disk file. The solution is to use a second, clean kernel to perform the capture. This is the core idea behind kdump.

The process relies on kexec, a feature that allows one kernel to boot directly into another, bypassing the BIOS/firmware and hardware initialization. kdump uses kexec in a specific way:

1. Reservation: When the primary (or "production") kernel boots, it reserves a small, protected region of RAM using the crashkernel boot parameter.
2. Pre-loading: A second, minimal "capture" kernel and a special initramfs are loaded into this reserved memory. This capture kernel sits dormant while the production kernel runs the system.
3. Panic!: When the production kernel panics, instead of halting, it triggers a kexec into the pre-loaded capture kernel.
4. Capture: The capture kernel boots in a clean state. Its main job is to access the memory of the crashed production kernel (which is now just static data in RAM) and save it to a file. This file is often called vmcore.
5. Reboot: Once the dump is saved, the capture kernel reboots the machine back into the normal production kernel.

This two-kernel dance is the foundation of reliable crash dumping on Linux.

Let's look at the components and flow in more detail.

2. Configuring Your System for Kdump

Setting up kdump involves a clear, sequential process of configuring the kernel, bootloader, and user-space services. Let's walk through it step-by-step.

Step 1: Kernel Configuration

First, you must ensure your kernel is built with the necessary features. For most modern distributions, these are enabled by default, but if you're building a custom kernel for an embedded device, you must verify them.

Here's a summary of the essential kernel options:

• CONFIG_KEXEC=y: Enables the core kexec system call.
• CONFIG_CRASH_DUMP=y: Enables the special kexec_load() functionality for a crash kernel.
• CONFIG_PROC_VMCORE=y: This is crucial for the capture kernel. It creates the /proc/vmcore pseudo-file, which provides an ELF-formatted view into the old kernel's memory.
• CONFIG_DEBUG_INFO=y: Just as with KGDB, this includes DWARF debugging symbols in the vmlinux file, which is essential for analyzing the vmcore later.
• CONFIG_MAGIC_SYSRQ=y: Allows you to trigger a crash manually for testing purposes.

Step 2: Reserving Memory with crashkernel

This is the most critical step. You must tell the production kernel to reserve memory for the capture kernel at boot time. This is done via a kernel command-line parameter in your bootloader configuration (e.g., grub.cfg).

The parameter is crashkernel=size. For example, crashkernel=256M.

After modifying your bootloader configuration and rebooting, you can verify that the memory was reserved:

# Check the kernel command line that was actually used
dmesg | grep "Command line"




# Look for the kernel message confirming the reservation
dmesg | grep "Reserving"



# Expected output: [ 0.000000] Reserving 256MB of memory at <address> for crashkernel




# You can also inspect the physical memory map
cat /proc/iomem | grep "Crash kernel"

Step 3: Installing and Configuring User-space Tools

With the kernel and bootloader prepared, the final step is to configure the user-space kdump service. This is typically handled by a package like kdump-tools (on Debian/Ubuntu) or kexec-tools (on Fedora/RHEL).

# On Debian/Ubuntu
sudo apt-get install kdump-tools crash




# On Fedora/RHEL
sudo dnf install kexec-tools crash

The main configuration file is usually /etc/kdump.conf (or /etc/sysconfig/kdump on older systems). This file tells the kdump service what to do when it runs inside the capture kernel environment.

A minimal /etc/kdump.conf might look like this:

# Path to save the crash dump
path /var/crash




# Use makedumpfile to compress the data and exclude unused pages
core_collector makedumpfile -l --message-level 1 -d 31




# Action to take after saving the dump
default reboot

After configuring, enable and start the service:

sudo systemctl enable kdump.service
sudo systemctl start kdump.service

3. Verifying and Testing the Kdump Setup

How do you know if it's working? First, you can check the status of the kdump service and see if the crash kernel is loaded.

Key verification commands:

• systemctl status kdump.service: Should show the service is active.
• cat /sys/kernel/kexec_crash_loaded: Should return 1.
• kdump-config show (on Debian/Ubuntu): Gives a comprehensive status report.

The final test is to trigger a panic and see if a dump is captured. The safest way to do this is with the Magic SysRq key.

# First, ensure SysRq is enabled
echo 1 | sudo tee /proc/sys/kernel/sysrq




# Trigger a manual kernel panic
echo c | sudo tee /proc/sysrq-trigger

What to expect:

1. Your system will instantly freeze and display a kernel panic message on the console.
2. The kexec mechanism will boot into the capture kernel. You may see boot messages from this second kernel.
3. The capture kernel will run the kdump scripts, which will use makedumpfile to save the vmcore to the path you specified in /etc/kdump.conf.
4. The machine will reboot.
5. After rebooting, you can navigate to /var/crash (or your configured path) and find a new directory named with the date and time of the crash, containing the vmcore file.

Congratulations! You have successfully captured a kernel crash dump.

4. Interview Scenario

Interviewer: "We have a fleet of embedded devices in the field that are mission-critical. Recently, a few of them have started to reboot unexpectedly. We suspect a kernel panic. We cannot attach a live debugger. Your task is to provide a clear, step-by-step procedure for our field technicians to enable post-mortem debugging on these devices so they can capture the crash information for you to analyze. What procedure do you give them?"

Conclusion

You have now bridged the gap between live debugging and post-mortem analysis. By configuring kdump, you've created a safety net that ensures even the most elusive crashes leave behind the evidence needed for diagnosis.

Key Takeaways:

• Kdump uses a two-kernel approach: A small, pre-loaded capture kernel boots upon the production kernel's panic to safely save its memory.
• The crashkernel= boot parameter is non-negotiable: It carves out the protected memory required for the capture kernel to live in.
• Configuration involves three stages: Enabling kernel CONFIG options, setting the crashkernel= boot parameter, and configuring the user-space kdump service.
• Testing is crucial: Always trigger a test panic with sysrq to ensure your configuration works before deploying it to a critical system.

Preview of the Next Lesson:

We've successfully captured a vmcore file. But this file is a raw, multi-gigabyte binary dump of memory. How do we turn it into useful information? In our next lesson, "Analyze a kernel crash dump using the 'crash' utility to perform a post-mortem root cause analysis," we will learn how to load this vmcore file into the powerful crash utility and begin dissecting it to find the bug that caused the panic.