Debugging Multi-threaded Programs with GDB

A Practical Scenario: The Runaway Thread

The best way to learn a debugger is to use it on a broken program. We're going to start with a classic and frustrating scenario: a program that is consuming 100% of a CPU core for no apparent reason.

We'll use an excellent, self-contained example from an article on Medium. First, let's get the code.

With the program running, open a new terminal. You won't see any output, but as the article points out, something is wrong. Let's use top to investigate.

1. Run top in your new terminal. You should see test.out at or near the top of the list, consuming close to 100% CPU. Note its Process ID (PID).
2. Press q to exit top.
3. Now, run top -H -p <PID>, replacing <PID> with the process ID you just noted. The -H flag tells top to show individual threads.

You'll see a list of threads within your process. One of them will be consuming 99.9% CPU, while the others are idle. The ID shown in the PID column for this runaway thread is actually its Lightweight Process (LWP) ID. This OS-level identifier is our first clue.

Attaching GDB and Finding Your Thread

Now that we've identified the problematic process and thread from the outside, it's time to go inside with GDB. We can attach the debugger directly to the already-running process.

In the same terminal, run sudo gdb ./test.out <PID>. You may need to enter your password. GDB will attach to the process and halt all of its threads.

Our first command inside GDB is the most important one for multithreaded debugging: info threads.

(gdb) info threads

This command lists all threads in the program.

Look at the output in your own GDB session. You will see several columns:

• ID: GDB's internal ID for the thread. This is the number you'll use in other GDB commands. An asterisk * marks the currently active thread.
• Target ID: This shows the system-level details, including the LWP ID we found with top -H.
• Frame: This tells you where the thread is currently paused—its current function and line number.

Your task: Match the LWP ID of the runaway thread from top -H to the Target Id column in info threads. This tells you which GDB thread ID (the number in the first column) corresponds to your CPU-hungry thread.

Switching Contexts and Inspecting State

The info threads output from the example program likely shows one thread spinning in thread_func while the others are sleeping in __GI___clock_nanosleep. Let's focus on the busy thread.

To switch GDB's context to a different thread, use the thread command with its GDB ID. For example, if the runaway thread is ID 6:

(gdb) thread 6
Switching to thread 6 (Thread 0x7913a75fa6c0 (LWP 45748) "test.out")
#0  0x00000000004011b0 in thread_func (p=0x3c75a8e0) at main.c:27
27	        while(val) {}

GDB confirms the switch and shows you the current stack frame for that thread.

Now that we are "in" the context of the problematic thread, we can use standard GDB commands to investigate:

• bt (or backtrace): Shows the complete call stack for the current thread.
• info locals: Displays the values of local variables in the current stack frame.
• p <variable> (or print <variable>): Prints the value of a specific variable.

In the test.out example, running bt on the busy thread will show it's in thread_func. Using info locals will reveal the variables inside that function, and you'll likely see that val is non-zero, causing the while(val) loop to spin forever. This pinpoints the bug.

This process—identify, attach, list, switch, inspect—is the fundamental workflow for debugging threads in GDB.

To learn more about the core commands from a different perspective, the following video provides a concise and clear demonstration.

Controlling the Chaos: The Scheduler Lock

By default, when you step through code (e.g., with next or step), GDB only controls the current thread. All other threads are allowed to run freely in the background. This can be chaotic; shared data can change under your feet, making it impossible to reason about the program's state.

To solve this, GDB provides the scheduler-locking feature.

The three modes for scheduler-locking are:

• off (default): Only the current thread is held, others run freely.
• on: Only the current thread is allowed to run. All other threads are frozen. This is perfect for isolating a thread's logic but risky if threads share locks.
• step: A hybrid. Other threads are frozen during next, step, etc., but run freely during commands like continue. This is often a safe and useful compromise.

Knowing how to lock the scheduler is an advanced GDB skill that gives you precise control when debugging complex interactions.

Conclusion

Congratulations! You have just learned the foundational toolset for debugging multithreaded C programs. While these commands are simple, they are incredibly powerful and form the basis of all advanced debugging techniques.

Key Takeaways:

• You can identify problematic threads from outside your program using tools like top -H to find their LWP ID.
• info threads is your main dashboard in GDB, listing all threads and mapping OS LWP IDs to GDB's internal IDs.
• You can switch GDB's focus using thread <ID> and then use standard commands like bt and print to inspect that specific thread's state.
• thread apply all bt gives you a complete snapshot of the program by showing the call stack for every thread at once.
• set scheduler-locking on gives you fine-grained control, allowing you to debug one thread in isolation by freezing all others.

Preview of the Next Lesson:
Now that you can see what threads are doing, the next logical step is to control where they stop. In the next lesson, we will learn how to set thread-specific breakpoints and navigate between thread backtraces in GDB to diagnose hangs, allowing you to precisely target and halt execution to capture the state you need to examine.