Debugging Multithreaded Application Hangs with GDB

Why Thread-Specific Breakpoints?

Imagine a program with one "writer" thread and dozens of "reader" threads all accessing the same function. If you set a normal breakpoint in that function, GDB will halt every time any thread hits it, creating a lot of noise. A thread-specific breakpoint allows you to say, "Only stop when the writer thread gets here." This surgical precision is key to efficient debugging.

The GDB syntax is a simple extension of the break command you already know. Let's see how it works.

The key command is break <location> thread <thread_id>, where <location> can be a line number (e.g., main.c:42) or a function name.

This video from Jacob Sorber also covers the same command, reinforcing the concept.

Here is a visual example of what you see in GDB when a thread-specific breakpoint is hit.

Diagnosing a Hang: A Practical Walkthrough

Now, let's apply these concepts to a classic multithreaded bug: a deadlock. We'll create a program that is guaranteed to hang, and then use our GDB skills to diagnose exactly why it's stuck.

Here is a simple C program where two threads attempt to acquire two mutexes in a conflicting order, causing a deadlock.

// deadlock.c
#include <stdio.h>
#include <pthread.h>
#include <unistd.h>

pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t mutex2 = PTHREAD_MUTEX_INITIALIZER;

void* thread1_func(void* arg) {
    printf("Thread 1: Trying to lock mutex1...\n");
    pthread_mutex_lock(&mutex1);
    printf("Thread 1: Locked mutex1.\n");

    // Give thread 2 a chance to run and lock mutex2
    sleep(1);

    printf("Thread 1: Trying to lock mutex2...\n");
    pthread_mutex_lock(&mutex2);
    printf("Thread 1: Locked mutex2.\n");

    pthread_mutex_unlock(&mutex2);
    pthread_mutex_unlock(&mutex1);
    return NULL;
}

void* thread2_func(void* arg) {
    printf("Thread 2: Trying to lock mutex2...\n");
    pthread_mutex_lock(&mutex2);
    printf("Thread 2: Locked mutex2.\n");

    // Give thread 1 a chance to run and lock mutex1
    sleep(1);

    printf("Thread 2: Trying to lock mutex1...\n");
    pthread_mutex_lock(&mutex1);
    printf("Thread 2: Locked mutex1.\n");

    pthread_mutex_unlock(&mutex1);
    pthread_mutex_unlock(&mutex2);
    return NULL;
}

int main() {
    pthread_t t1, t2;

    pthread_create(&t1, NULL, thread1_func, NULL);
    pthread_create(&t2, NULL, thread2_func, NULL);

    pthread_join(t1, NULL);
    pthread_join(t2, NULL);

    printf("Finished.\n");
    return 0;
}

1. Compile and Run: Save this code as deadlock.c. Compile it with debugging symbols and the pthreads library:
   gcc -g -o deadlock deadlock.c -pthread

2. Observe the Hang: Run ./deadlock. You will see output indicating both threads have locked one mutex, but the program will then hang indefinitely.

3. Debug with GDB:

   • Start the program in GDB: gdb ./deadlock
   • Type run and press Enter. The program will run and hang as before.
   • Press Ctrl-C to interrupt the program and return control to GDB.

4. Diagnose the Hang: Now we execute the core diagnostic pattern we discussed.

   • First, get a snapshot of all threads: thread apply all bt

You should see an output where one thread is stuck in pthread_mutex_lock trying to get mutex2, and the other is also stuck in pthread_mutex_lock trying to get mutex1. The backtraces (bt) clearly show the line numbers in deadlock.c where they are blocked. This output is the "smoking gun" for a deadlock.

Navigating Between Backtraces

The thread apply all bt command gives you the global view. To "navigate between backtraces," you simply switch to each thread involved and inspect its context more closely.

1. Use info threads to get the GDB IDs for thread1_func and thread2_func.
2. Switch to the first thread: thread <ID_of_thread1>
3. Examine its stack in detail: bt. The backtrace will show it's blocked on pthread_mutex_lock waiting for mutex2.
4. Switch to the second thread: thread <ID_of_thread2>
5. Examine its stack: bt. This will show it's blocked waiting for mutex1.

By switching between the threads and looking at their individual call stacks, you have effectively navigated the dependency chain of the deadlock and confirmed the root cause. You can now use this information to fix the code (e.g., by enforcing a consistent lock acquisition order).

Advanced Insight: Scheduler Locking and Deadlocks

In the previous lesson, we learned about set scheduler-locking on. This command freezes all other threads, which is great for isolating one thread's logic. However, it comes with a major caveat: you can accidentally cause a deadlock within the debugger.

This video provides a brilliant demonstration of this exact scenario and how to diagnose it using the techniques we've just practiced.

This scenario highlights the power and synergy of these commands. set scheduler-locking gives you control, but thread apply all bt is your essential tool for understanding the global state, especially when that control leads to unintended consequences like a debugger-induced hang.

Conclusion

You have now moved beyond basic inspection and into surgical control of multithreaded programs in GDB. Being able to set breakpoints on specific threads and quickly analyze the state of all threads in a hung program are indispensable skills for tackling complex concurrency bugs.

Key Takeaways:

• Use break <location> thread <thread_id> to set breakpoints that only stop a specific thread.
• The primary diagnostic pattern for a hung program is: Ctrl-C -> thread apply all bt.
• Navigating between backtraces involves using thread <ID> to switch contexts and bt to inspect the call stack of each relevant thread, revealing the chain of dependencies causing the hang.
• Be mindful that set scheduler-locking on can cause its own deadlocks, which you can then diagnose with thread apply all bt.

Preview of the Next Lesson:
Manually hunting for deadlocks and race conditions works, but it can be slow and requires you to know where to look. In our next lesson, we will explore Valgrind's Helgrind tool, a dynamic analysis tool that can automatically detect data races and other synchronization errors in your C program, often without any manual debugging effort.