Memory Barriers: Ensuring Correct Memory Ordering

Why We Need Barriers: ARM's Weak Memory Model

To maximize performance, modern processors like ARM do not guarantee that memory operations will be executed in the order they appear in your program. This is known as a weakly-ordered or relaxed memory model. The CPU might reorder non-dependent reads and writes, use store buffers that delay when a write becomes visible to other cores, and execute instructions speculatively.

For a single core, the processor ensures these tricks don't change the program's outcome. But in a multi-core system, where multiple cores ("observers") are looking at the same shared memory, this reordering can lead to subtle and catastrophic bugs.

The only ordering guarantees the ARM architecture provides by default are for dependent accesses (e.g., loading a pointer and then dereferencing it). For everything else, you must assume reordering can and will happen.

Let's watch a segment of a talk from an Amazon engineer that brilliantly illustrates this problem.

As the video clearly showed, relying on program order for synchronization between cores is a recipe for failure. We need to tell the processor explicitly when ordering matters. This is done with memory barrier instructions.

The Three Essential Tools: DMB, DSB, and ISB

The ARM architecture provides three main barrier instructions to enforce order and synchronization.

To summarize and expand on those definitions:

1. DMB - Data Memory Barrier: This is the "ordering" barrier. It ensures that all memory accesses (loads and stores) appearing in the code before the DMB are observed by other cores before any memory accesses after the DMB. It does not stall the CPU; it just prevents reordering of memory operations across the barrier. This is the most common barrier used for general-purpose synchronization, like in our spinlock.

2. DSB - Data Synchronization Barrier: This is a "completion" barrier. It is much stricter than DMB. A DSB will stall the execution of any further instructions (not just memory accesses) until all memory accesses before the DSB have fully completed. This is crucial when your code's logic depends not just on the order, but on the side effects of a memory write, such as interacting with a hardware peripheral.

3. ISB - Instruction Synchronization Barrier: This is the "pipeline flush" barrier. It's the strictest of all. An ISB flushes the processor's pipeline and prefetch buffers, causing all instructions following the ISB to be re-fetched from memory. This is necessary after operations that change the context of execution, such as modifying page tables, MPU settings, or writing new code to memory (self-modifying code).

Now, let's learn more about DSB and ISB from the same video.

Putting Barriers to Use: From Theory to Practice

Knowing the barriers is one thing; using them effectively is another. Using a DSB or ISB when a DMB would suffice can needlessly harm performance. The key is to use the minimally intrusive barrier that guarantees correctness.

DMB and Performance: Qualifiers

The DMB instruction can be made more granular (and thus more performant) using qualifiers. These qualifiers restrict the barrier's effect in two ways:

1. Shareability Domain: Specifies which observers the ordering applies to.

   • ISH (Inner Shareable): Typically the group of CPUs in a single cluster (e.g., your SMP Linux cores).
   • OSH (Outer Shareable): A wider domain, including other clusters or bus masters like a GPU.
   • NSH (Non-Shareable): Only applies to observers within the issuing core itself (e.g., instruction fetch vs. data access units).
   • SY (Full System): Applies to all observers in the entire system. This is the strongest and most expensive.

2. Access Type: Specifies which types of memory access are ordered.

   • ST (Store): Orders only stores before the barrier against stores after it (DMB ISHST means Inner-Shareable, Store-Store barrier).
   • LD (Load): Orders loads before the barrier against any access (load or store) after it.

By combining these, you can precisely state your ordering requirements. For the polled mailbox example, the writer only needs to order its two stores, and the reader only needs to order its two loads.

Using DMB ISHST is far more efficient than a full DMB SY, because the CPU doesn't have to worry about ordering any loads, and it only needs to ensure the ordering is visible to other cores in its own cluster, not every single device in the SoC. This is the level of detail expected in advanced driver development.

DSB for Hardware Synchronization (I/O & DMA)

Your work involves extensive interaction with peripherals. This is a classic use case for DSB. Imagine you're writing a UART driver. You write data to a DMA buffer in RAM, and then you write to a DMA controller's register to kick off the transfer.

1. STR r0, [BUFFER_ADDR] // Write data to RAM
2. STR r1, [DMA_START_REG] // Tell DMA controller to start

Because of store buffers, the write to the DMA register could become visible to the DMA controller before the data write has actually reached RAM. The DMA would then transfer garbage. A DMB is not sufficient here because it only guarantees ordering between the two accesses, not completion. The DMA controller is a separate bus master; we need to ensure the write to RAM is complete and visible across the system before we proceed. This requires a DSB.

The key takeaway is that when communicating with another master (like a peripheral), you often need to guarantee completion, not just ordering. This is DSB territory. In the Linux kernel, the wmb() (write memory barrier) macro often expands to a DSB for this exact reason when dealing with I/O.

ISB for Context-Altering Operations

Finally, ISB is used when the fundamental rules of execution change. A prime example from your domain is programming a Memory Protection Unit (MPU).

When you reconfigure MPU regions, you are changing memory access permissions. After writing to the MPU configuration registers, the CPU's pipeline might still contain instructions that were fetched before the change, under the old rules. Executing these could violate the new memory protection scheme.

The correct sequence is:

1. Write to MPU registers.
2. Execute a DSB to ensure those writes to the MPU (which is a memory-mapped peripheral) are complete.
3. Execute an ISB to flush the pipeline and ensure that any subsequent instruction fetches happen with the new MPU settings in effect.

Interview Corner

Mastering this topic will set you apart in a technical interview.

Q1: What is a memory barrier and why is it needed on ARM processors?
A: A memory barrier is an instruction that enforces an ordering constraint on memory operations. It's necessary on ARM because it has a weakly-ordered memory model, meaning the processor can reorder memory accesses for performance. In a multi-core system, this can break synchronization logic that relies on program order. Barriers are used to explicitly tell the processor "do not reorder accesses across this point."

Q2: Explain the difference between DMB, DSB, and ISB. Give a brief example use case for each.
A:

• DMB (Data Memory Barrier) enforces ordering. It ensures all memory accesses before it are observed before accesses after it, but it doesn't stop the CPU. It's used for synchronizing shared data between cores, like in a spinlock implementation.
• DSB (Data Synchronization Barrier) enforces completion. It stalls the CPU until all prior memory accesses are complete. It's used when synchronizing with hardware state, for example, ensuring data is written to a buffer in RAM before writing to a DMA controller's register to start a transfer.
• ISB (Instruction Synchronization Barrier) enforces context synchronization. It flushes the CPU pipeline and instruction caches. It's used after an operation that changes the execution context, like updating MPU settings, modifying page tables, or jumping to newly written code in memory.

Q3: What is the purpose of barrier qualifiers like ishst? Why not just use a full system barrier (dmb sy) everywhere?
A: Qualifiers make barriers more granular and performant. ishst means "Inner Shareable, Store-Store," which limits the barrier's scope. It only orders store operations against other stores, and only guarantees this ordering is visible within the local CPU cluster (the inner shareable domain). A full system barrier (dmb sy) is a "sledgehammer" approach—it orders all access types (loads and stores) and forces that ordering to be visible to every single component in the entire SoC. While always correct, it's unnecessarily expensive and hurts performance. Using the minimally intrusive barrier (ishst in this case) that satisfies the specific requirement of the algorithm is key to writing efficient low-level code.

Conclusion

Today, you've taken a deep dive into one of the most complex but crucial aspects of modern embedded programming. We've moved beyond simply knowing a barrier is needed to understanding why it's needed and which specific barrier to use.

Key Takeaways:

• ARM's weakly-ordered memory model allows reordering of memory accesses for performance, which can break multi-core synchronization.
• DMB is an ordering barrier, used to sequence memory accesses between observers (e.g., two CPU cores).
• DSB is a completion barrier, used to ensure memory operations are finished before the CPU proceeds, which is vital for hardware interaction.
• ISB is a pipeline flush barrier, used to synchronize the execution context itself.
• Using qualifiers like ish and st allows for minimally intrusive barriers, which is critical for balancing correctness and performance.

Next Lesson Preview:
Barriers ensure writes become visible in the correct order, but caches introduce another problem: where is the "memory" that other cores see? A write might be sitting in one core's local L1 cache, invisible to another core. In our next lesson, we will explore cache coherency issues in multi-core systems and the role of hardware coherency mechanisms like the MESI protocol to solve them.