Multi-core Debugging with JTAG: Synchronized Control

1. The Challenge: Why Multi-Core Debugging is Different

In a single-core system, debugging is relatively straightforward. You can stop the core at a breakpoint, and the entire world freezes with it. You can inspect memory and registers, confident that you are seeing a consistent snapshot of the system's state.

This assumption breaks down in a multi-core environment.

As the article explains, if you stop only one core, the other cores continue running. By the time you manually halt them, the overall system state has changed, and the context of the original breakpoint is lost. The root cause of your bug may have been overwritten, or the system may now be in an entirely new, unrelated state.

This leads to the core requirement of multi-core debugging: synchronized run control. You need the ability to start, stop, and step multiple cores (or a specific group of cores) in unison, as if they were a single entity.

2. Multi-Core Debugging Architectures

Before configuring the debugger, you need to understand the model it uses. JTAG debuggers like Lauterbach TRACE32 typically support two main multi-core debugging paradigms: Asymmetric Multiprocessing (AMP) and Symmetric Multiprocessing (SMP). Your experience with AUTOSAR communication stacks and gateway development has likely exposed you to AMP systems, where different cores might run different operating systems or bare-metal applications.

As the image shows:

• AMP Debugging: You launch a separate instance of the TRACE32 software for each core you want to debug. Each instance is configured for its specific core and application. This is the most common model for heterogeneous systems (e.g., Cortex-A + Cortex-M) or homogeneous systems running different software stacks.
• SMP Debugging: A single TRACE32 instance is launched with OS-awareness (e.g., for Linux or an SMP-aware RTOS). The debugger understands that the OS can schedule threads on any core and presents a unified view of the system.

For embedded driver development, you will most frequently encounter the AMP model, where you need to explicitly manage the debug session for each core.

3. Configuring a Multi-Core Debug Session

Setting up an AMP debug session involves three main steps:

1. Hardware Connection: Connect the Lauterbach PowerDebug module to the target's JTAG port.
2. Configuration Files (config.t32): Prepare configuration files that tell each TRACE32 instance how to connect to its specific core.
3. Synchronization Script (.cmm): Write a PRACTICE script to launch the TRACE32 instances and configure the synchronization between them.

The following guide is the definitive source for this process. We'll walk through the key parts.

Summary of Configuration

To set up a debug session for a dual-core AMP system, you would typically have:

• config_core0.t32:

  PBI=
    CORE=1      // This instance targets the first core
    USB
  
  IC=
    NETASSIST
    PORT=20001  // Opens a port for Core 1's debugger to connect to
  

• config_core1.t32:

  PBI=
    CORE=2      // This instance targets the second core
    USB
  
  IC=
    NETASSIST
    PORT=20002  // Opens a port for Core 0's debugger to connect to
  

• A master script (startup.cmm) to launch and synchronize everything.

4. Implementing Synchronized Execution Control

With the instances configured to communicate, you can now establish synchronized run control. This is done using the SYnch command group in a PRACTICE script.

One core is designated the master, and the others are slaves.

• When the master starts, all slaves with SlaveGo enabled will also start.
• When the master breaks (hits a breakpoint, is manually halted), all slaves with SlaveBrk enabled will also break.

Here is a simplified version of a startup script for a dual-core system, which you would run from the TRACE32 instance for Core 0:

// startup_core0.cmm

// 1. Launch the TRACE32 instance for Core 1
OS.Task t32marm -c config_core1.t32 -s startup_core1.cmm

// 2. Wait for Core 1's debugger to be ready (it will have opened port 20002)
InterCom.WAIT localhost:20002

// 3. Configure this instance (Core 0) as the MASTER
SYnch.RESet
SYnch.Connect localhost:20002  // Connect to Core 1's debugger
SYnch.MasterGo ON
SYnch.MasterBrk ON

// 4. Configure the other instance (Core 1) as the SLAVE
//    Use InterCom to send commands to the other TRACE32 instance
InterCom.execute localhost:20002 "SYnch.RESet"
InterCom.execute localhost:20002 "SYnch.Connect localhost:20001" // Connect back to me
InterCom.execute localhost:20002 "SYnch.SlaveGo ON"
InterCom.execute localhost:20002 "SYnch.SlaveBrk ON"

// Now, when you press "Go" in the Core 0 window, Core 1 will also run.
// When a breakpoint is hit on either core, both will stop.

5. Analyzing a Multi-Core Session

Now that you can synchronously control the cores, you can effectively analyze their behavior. A multi-core debug session provides a window into all concurrent activities.

With this setup, you can perform powerful debugging actions:

• Global Breakpoints: When a breakpoint is hit on Core 0, Core 1 stops simultaneously. You can then inspect the call stack, registers, and memory of both cores to understand the exact state of the entire system at the moment the breakpoint was hit.
• Cross-Core Analysis: Imagine debugging the IPC mechanism from our last lesson. You can set a breakpoint on Core 0 just after it triggers the SGI, and another breakpoint in the SGI's ISR on Core 1. Running the system will instantly tell you if the interrupt is being delivered and handled.
• Conditional, Core-Specific Breakpoints: You can set advanced breakpoints that only trigger for a specific task or thread on a specific core.

Interview Corner

Q1: You suspect a deadlock where Core 0 is waiting for a resource held by Core 1, and Core 1 is waiting for a resource held by Core 0. How would you use a JTAG debugger like Lauterbach to confirm this?
A: I would first establish a synchronized multi-core debug session. Then, I would manually halt the master core. Due to synchronization, the slave core would also halt. I would then inspect the call stack of both cores. If Core 0 is stuck in a while() loop or spinlock waiting for a flag A, I would check the code to see that this flag is supposed to be released by Core 1. Then I'd examine Core 1's call stack. If it's similarly stuck waiting for a flag B that is supposed to be released by Core 0, I have confirmed the deadlock. The synchronized halt gives me the critical, simultaneous snapshot of both cores' states needed to prove the circular dependency.

Q2: What is "synchronous run control," and why is it essential for multi-core debugging?
A: Synchronous run control is the ability of a debugger to start, stop, and step multiple processor cores in unison. It's essential because multi-core bugs, like race conditions and deadlocks, depend on the precise timing and interaction between cores. Without it, if you stop one core at a breakpoint, the others continue running, changing the system state and making it impossible to analyze the exact conditions that led to the bug. Synchronous control freezes the entire system at a point in time, preserving the state of all cores for a consistent and meaningful analysis.

Q3: You're setting up a Lauterbach session for a dual-core AMP system. What are the key settings you need in your configuration files (.t32) and your startup script (.cmm)?
A:

• In the configuration files (config_core0.t32, config_core1.t32):
  • PBI=CORE=<n>: Each file needs to specify which core its debugger instance will attach to (e.g., CORE=1 for the first, CORE=2 for the second).
  • IC=NETASSIST PORT=<port>: Each instance must open a unique TCP/IP port for inter-debugger communication. For example, Core 0's config might use PORT=20001 and Core 1's might use PORT=20002.
• In the startup script (startup.cmm):
  • The script for the master core (e.g., Core 0) needs to launch the slave's debugger instance, typically using the OS.Task command.
  • It must use SYnch.Connect to connect to the port opened by the slave debugger (e.g., SYnch.Connect localhost:20002).
  • It configures itself as master using SYnch.MasterGo ON and SYnch.MasterBrk ON.
  • It uses InterCom.execute commands to send configuration commands to the slave instance, telling it to connect back (SYnch.Connect localhost:20001) and to act as a slave (SYnch.SlaveGo ON, SYnch.SlaveBrk ON).

Conclusion

This lesson concludes our deep dive into multi-core programming. We've journeyed from the hardware foundations of cache coherency and memory barriers to the software constructs for synchronization and communication. Today, you learned how to master the tool that ties it all together: the multi-core debugger.

Key Takeaways:

• Multi-core debugging requires synchronized run control to capture a consistent system state.
• In an AMP system, you typically run one debugger instance per core.
• Configuration involves telling each instance which core to attach to (PBI=CORE=n) and opening a communication channel (IC=NETASSIST PORT=...).
• Synchronization is achieved via a master/slave model configured with the SYnch and InterCom commands in a PRACTICE script.
• With a synchronized session, you can use global breakpoints and conditional watchpoints to effectively find and analyze complex multi-core bugs like race conditions and deadlocks.

Next Module Preview:
You have now built a formidable understanding of the low-level workings of modern ARM cores. We will now move up one layer of abstraction. The next module, AUTOSAR MCAL Development, will show you how these hardware features are controlled and exposed through the standardized interfaces of the AUTOSAR Microcontroller Abstraction Layer. Your knowledge of registers, interrupts, and peripherals will provide an excellent foundation for understanding and implementing MCAL drivers like Port, Dio, and Mcu.