Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 52

Optimizing Multi-Task Application Performance

Recommended resources

Hello! Welcome to the final lesson of our course on C++ for the ESP32.

In our last session, we equipped ourselves with a powerful set of analysis tools. You learned how to use Segger SystemView to visualize task execution, uxTaskGetStackHighWaterMark to monitor memory, and vTaskGetRunTimeStats to check CPU load. These tools act as our system's "dashboard," telling us what is happening inside.

Today, we'll learn how to act on that information. This lesson focuses on the final and most critical step in embedded development: debugging and optimizing our multi-task application to meet performance requirements for response time and stability. We will move from being observers to being surgeons, precisely targeting and fixing issues. We will adopt an engineering mindset: measure the problem, identify the root cause, apply a solution, and measure again to verify the fix.

By the end of this lesson, you will have the skills to tackle some of the most challenging problems in real-time systems and ensure your applications are both robust and performant.

1. Understanding the Enemy: Common RTOS Bugs

In a multi-tasking environment, bugs are often more complex than simple logic errors. They arise from the interactions between tasks, the scheduler, and shared resources. Before we can fix them, we need to know what they look like.

Debugging FreeRTOS Applications

The article 'Debugging FreeRTOS Applications' provides an excellent overview of the typical challenges you'll face. Pay close attention to the table of common problems.

Read the section 'Common Debugging Challenges in RTOS Applications'. Focus on understanding the description and symptoms of Race Conditions, Deadlocks, Priority Inversion, and Stack Overflows. This will help you recognize these issues when you see their symptoms in your own projects.

These issues—like corrupted data that appears randomly or a system that mysteriously freezes—can be nearly impossible to diagnose with simple printf statements. For these, we need a more powerful tool.


2. Interactive Debugging with a Hardware Debugger (JTAG)

The most powerful weapon in your debugging arsenal is a hardware debugger. Using a JTAG probe (like the ESP-PROG), you can gain complete control over the ESP32's CPU. This allows you to pause the execution of your program at any point, inspect the state of variables and memory, and step through the code line by line—even across different tasks.

While we used JTAG in the last lesson to stream trace data for SystemView, its primary power lies in interactive debugging.

Setting Up and Using the Debugger

Setting up a JTAG debugger involves a few steps: connecting the hardware, installing drivers, and configuring your project in VS Code. The process can seem intricate, but it's a one-time setup that pays huge dividends.

ESP32 - DEBUGGING your ESP-IDF code using JTAG [VS CODE]

The video 'ESP32 - DEBUGGING your ESP-IDF code using JTAG' provides a clear, step-by-step guide to getting a JTAG debugger working from scratch.

Please watch from 0:46 to 14:00. This is a comprehensive walkthrough that covers everything you need. Focus on these key stages: Introduction (0:46 - 2:56): Understand the limitations of serial printing and the power of JTAG for setting breakpoints and controlling execution. Hardware & Driver Setup (3:22 - 7:37): See how to connect an ESP-PROG and use the Zadig tool to install the correct Windows driver. Software Configuration (7:37 - 14:00): This is the most critical part. Pay close attention to how openocd is launched and how the launch.json and tasks.json files are configured in VS Code. The speaker gives an essential tip about using a flag to tell OpenOCD you are debugging a single thread at a time (--command "set multithreaded off"), which dramatically improves debugger stability. Although counter-intuitive for an RTOS, it prevents the debugger from getting confused by multiple concurrent tasks.

Once set up, you can use the VS Code interface to:

  • Set Breakpoints: Click in the margin next to a line of code to pause execution when it's reached.
  • Inspect Variables: View the values of local and global variables in the "Variables" panel.
  • Examine the Call Stack: See the sequence of function calls that led to the current point. In an RTOS context, you can even switch between tasks (threads) to see the call stack of each one.
  • Control Execution: Use the debug controls (Continue, Step Over, Step Into) to navigate your code.

A typical launch.json configuration for ESP-IDF. This file tells VS Code how to launch GDB, connect to OpenOCD, and what initial commands to run to set up the debugging session.

Test your understanding!

You are debugging a function that processes sensor data. You notice that adding ESP_LOGI statements to print variable values makes a bug disappear. When you remove the logs, the bug returns. Which debugging tool should you use to investigate this, and why?

Show answer

This is a classic "Heisenbug"—a bug that changes its behavior when you try to observe it. The timing introduced by the logging statements is affecting the program's execution, likely masking a race condition or another timing-sensitive issue.

The best tool here is the JTAG hardware debugger. It allows you to halt the CPU and inspect the state of variables and memory without adding extra code or significantly altering the timing of the application (at least, not until a breakpoint is hit). This gives you a much more accurate snapshot of the system's state at the moment the bug occurs.


3. Optimizing for Performance and Stability

Debugging is about fixing what's broken. Optimization is about making what works, work better. For our system, this means ensuring the motor control is highly responsive and the entire application is stable under all conditions.

3.1. Achieving Low Latency with Task Priorities and Core Affinity

In our project, the motor_control_task must respond to commands immediately. It should not be delayed by the lower-priority temp_sensor_task. We've already set its priority higher, but we can do even better by leveraging the ESP32's dual-core architecture.

Most of the ESP-IDF background tasks (like Wi-Fi, Bluetooth, and event loops) are configured by default to run on Core 0. This gives us a golden opportunity: we can run our most time-critical application code on Core 1 with minimal interference.

Maximizing Execution Speed - ESP32 - — ESP-IDF ...

The ESP-IDF documentation on 'Maximizing Execution Speed' contains a crucial section on task priorities that explains this strategy.

Read the sections 'Task Priorities', 'Built-In Task Priorities', and 'Choosing application task priorities'. Focus on: The list of built-in tasks and their default priorities. The recommendation to pin high-priority application tasks to Core 1 to avoid being preempted by system tasks. The recommended priority levels to use to ensure your task preempts other tasks like the lwIP TCP/IP stack if needed.

Strategy for our project:

  1. Motor Control Task: Pin to Core 1 with a high priority (e.g., 19 or 20). This ensures it runs almost immediately when it receives a command, as there's very little competition on Core 1.
  2. Sensor Task: Can be left unpinned or pinned to Core 0 with a lower priority (e.g., 5). Its timing is not as critical.
  3. Other Application Tasks: Any other non-critical tasks (like a UI or logging task) should have low priorities.

You can set the core affinity when you create the task using xTaskCreatePinnedToCore().

// Create the high-priority motor task on Core 1
xTaskCreatePinnedToCore(
    motor_control_task,     // Task function
    "MotorControl",         // Task name
    4096,                   // Stack size
    NULL,                   // Task parameters
    20,                     // Priority
    &motor_task_handle,     // Task handle
    1                       // Core ID
);

// Create the lower-priority sensor task on Core 0
xTaskCreatePinnedToCore(
    temp_sensor_task,       // Task function
    "TempSensor",           // Task name
    4096,                   // Stack size
    NULL,                   // Task parameters
    5,                      // Priority
    &sensor_task_handle,    // Task handle
    0                       // Core ID
);

3.2. Ensuring Stability: Detecting and Fixing Stack Overflows

One of the most common causes of instability is a stack overflow. In the last lesson, you learned to check the stack high water mark. What do you do when it's dangerously low?

The solution is a two-step process:

  1. Detect the overflow: Enable stack checking in menuconfig.
  2. Implement a hook function: This function will be called by FreeRTOS when an overflow occurs, allowing you to get crucial debug information before the system crashes.

Debugging FreeRTOS Applications

Let's walk through a complete example of how to enable stack checking and what to do when an overflow is detected. This practical example will show you the full 'detect, diagnose, fix' cycle.

Read and study 'Example 2: Detecting Stack Overflow'. Follow the steps provided: See how to enable stack checking in menuconfig. Examine the vApplicationStackOverflowHook function. This is what runs when the overflow happens, and it's your chance to log the name of the offending task. Analyze the stack_overflow_task to understand how a large local variable or deep recursion can cause an overflow. Review the expected output to see what a stack overflow crash log looks like.

Once you've identified which task is overflowing, the fix is usually either to increase its stack size in xTaskCreate() or to optimize the function by reducing the size of local variables or avoiding deep recursion.

3.3. General Optimization Checklist

Beyond task priorities and stack management, you can improve overall performance with a few key settings.

Maximizing Execution Speed - ESP32 - — ESP-IDF ...

The 'Maximizing Execution Speed' document has a great list of general-purpose optimizations. You can think of this as a quick-win checklist for performance.

Read the sections 'Improving Overall Speed' and 'Targeted Optimizations'. You don't need to memorize every detail, but be aware of these key levers you can pull: Flash Speed and Mode: Increasing flash frequency (CONFIG_ESPTOOLPY_FLASHFREQ) and using Quad I/O mode (CONFIG_ESPTOOLPY_FLASHMODE) can significantly speed up code execution from flash. Compiler Optimization: Setting CONFIG_COMPILER_OPTIMIZATION to 'Optimize for performance (-O2)' tells the compiler to prioritize speed. Placing Code in IRAM: For functions that are executed very frequently or have strict timing requirements (like ISRs or control loops), placing them in IRAM using the IRAM_ATTR macro can eliminate delays from flash cache misses. This is a powerful technique for mechanical control applications.


4. Post-Mortem Debugging with Core Dumps

What happens if your device crashes in the field, far away from your JTAG debugger? For these situations, ESP-IDF provides a core dump mechanism. When a fatal error occurs, the ESP32 can save the state of the CPU and the call stacks of all tasks to flash memory.

You can later retrieve this core dump and use a script (espcoredump.py) on your development machine to analyze it. This "post-mortem" analysis will show you the exact state of the system at the moment of the crash, including the function and line number that caused the fault.

Debugging FreeRTOS Applications

Let's briefly cover the concept and utility of core dumps.

First, read the description of 'Core Dumps' in the 'Debugging Tools and Techniques' section to understand what they are and why they're useful. Then, look at 'Exercise 5: Core Dump Analysis' to see the high-level workflow of how you would trigger a crash, capture the dump, and analyze it with the espcoredump.py tool. This is an advanced but essential technique for creating truly robust products.


Conclusion and Course Wrap-up

Congratulations on completing not just this lesson, but the entire course! You have journeyed from the fundamentals of C++ and object-oriented programming to the intricate details of building, analyzing, and optimizing a complex, multi-tasking embedded system on the ESP32.

Key Takeaways from this Lesson:

  • Real-time systems introduce unique bugs like deadlocks and race conditions that require advanced debugging techniques.
  • Interactive JTAG debugging is the most powerful tool for finding the root cause of complex software faults.
  • System performance can be drastically improved by strategic task prioritization and core affinity, leveraging the ESP32's dual-core architecture.
  • Ensuring stability involves proactively using tools like stack overflow detection and core dumps to catch and diagnose errors.
  • Optimization is an iterative process: measure, identify, fix, and re-measure.

You now possess a strong, practical foundation for developing sophisticated applications with C++ on the ESP32. You understand the hardware peripherals, the real-time operating system, and the tools needed to build reliable and performant products. The world of advanced embedded systems is now open to you. Happy coding