Memory Barriers: Compiler, Volatile, and Hardware

1. The Two Beasts of Reordering: Compiler and CPU

When you write C code, you intuitively expect the operations to happen in the order you wrote them. However, for performance, both the compiler and the CPU are allowed to reorder your instructions, as long as they don't change the outcome for a simple, single-threaded program.

1. Compiler Reordering: The compiler reorganizes your code during compilation to generate more efficient assembly.
2. CPU Reordering: A modern processor executes instructions out-of-order to keep its execution units busy, hiding memory latency. This is especially prevalent on weakly-ordered architectures like ARM.

In a concurrent system (multi-core, or with interrupts), this reordering can break your code in subtle and hard-to-debug ways. To regain control, we need to issue specific commands—barriers—to both the compiler and the CPU.

2. Taming the Compiler: volatile and Compiler Barriers

Let's start with the compiler. Its goal is to produce fast code. If it sees an opportunity to rearrange or even eliminate memory accesses without changing the single-threaded program logic, it will take it.

The volatile Keyword: A Limited Tool

A common tool to control the compiler is the volatile keyword. Its C standard definition is narrow but precise: every access to a volatile-qualified object in the abstract C machine must correspond to an actual load or store on the real machine.

This is essential for two main scenarios:

• Memory-Mapped I/O (MMIO): Accessing a hardware register whose value can change at any time.
• Variables shared with an ISR: Ensuring the main code sees changes made by an interrupt handler.

The Critical Limitation of volatile

Here is the key interview question: Is volatile a memory barrier? No.

volatile only guarantees that accesses to that specific volatile variable are not optimized away. It provides no ordering guarantees with respect to other non-volatile memory accesses. The compiler is free to move a non-volatile store or load from before a volatile access to after it, or vice versa.

This is one of the most misunderstood concepts in C. Let's look at a canonical example of where this fails.

Compiler Barriers: The Real Solution for Ordering

As you just read, to prevent the compiler from reordering memory operations around a specific point, we need an explicit compiler barrier. In GCC and compatible compilers, this is:

asm volatile ("" ::: "memory");

This inline assembly statement tells the compiler:

1. It contains no actual instructions ("").
2. It is volatile, so it must not be removed.
3. It "clobbers" memory ("memory"). This is the key part. It tells the compiler to assume that this operation could read or write any location in memory.

This forces the compiler to:

• Commit any modified register values to memory before the barrier.
• Not move any memory accesses from before the barrier to after it.
• Not move any memory accesses from after the barrier to before it.

For a more comprehensive look at compiler reordering and barriers, the following article provides another excellent perspective, including the evolution of volatile usage in the Linux kernel to macros like READ_ONCE.

3. Taming the CPU: Hardware Memory Barriers

Now, assume you've used compiler barriers correctly. Your compiled assembly instructions are in the exact order you need. You're still not safe! A modern CPU, especially a weakly-ordered one like ARM, can still execute these instructions out of order.

The most intuitive reason for this is the store buffer. To avoid stalling on slow memory writes, a CPU core will often write data into a small, fast buffer (the store buffer) and then continue executing subsequent instructions. This means a store might become visible to other cores much later than a subsequent load from the same core.

On strongly-ordered architectures like x86, this store-load reordering is one of the few reorderings allowed. On weakly-ordered architectures like ARM, the rules are much more relaxed—almost anything can be reordered unless there is a direct data dependency.

This is where hardware memory barriers come in. These are actual CPU instructions that force the processor to enforce an ordering on its memory operations.

The following video is a fantastic deep dive into ARM's memory model, presented by an expert from Amazon. This is exactly the level of detail expected in a senior interview at a company like ARM or Qualcomm.

Types of Hardware Barriers

As you saw in the video, hardware barriers are not one-size-fits-all. To maximize performance, architectures provide different types of barriers for different ordering needs.

On ARM, the main barrier instructions are:

• DMB (Data Memory Barrier): Ensures that all memory accesses before the DMB are observed before any memory accesses after the DMB. It doesn't stop execution, but it enforces ordering of memory operations.
• DSB (Data Synchronization Barrier): Stricter than DMB. It blocks execution of any instructions (not just memory accesses) until all memory accesses before the DSB are complete. You need this when you're changing system state, like page tables, and need to ensure the change is complete before proceeding.
• ISB (Instruction Synchronization Barrier): The strictest barrier. It flushes the pipeline and prefetch buffers, causing instructions after the ISB to be re-fetched from memory. This is necessary after modifying self-modifying code or changing MMU settings that affect instruction fetches.

These barriers can be further qualified by the types of accesses they order (e.g., ST for store) and the group of observers they apply to (e.g., ISH for Inner Shareable, i.e., other cores in the same cluster).

4. Summary: The Three Layers of Ordering

Let's synthesize everything into a clear comparison. This is your key takeaway for an interview.

In short:

• Use volatile for MMIO or variables shared with an ISR to ensure visibility.
• Use a compiler barrier to enforce ordering between accesses to different variables at compile time.
• Use a hardware barrier to enforce ordering at runtime on the CPU itself.

In the Linux kernel, a full memory barrier like smp_mb() includes both a compiler barrier and a hardware barrier to provide a complete ordering guarantee against both threats.

Conclusion

You have now delved into the core of memory ordering in concurrent systems. You've distinguished between the reordering performed by the compiler and the CPU, and you've learned about the specific tools used to control each one: volatile, compiler barriers, and hardware barriers. Understanding this hierarchy is non-negotiable for writing correct, high-performance, low-level code.

Key Takeaways:

• volatile is not a memory barrier. It only prevents compiler optimizations on accesses to a single variable.
• Compiler barriers (asm volatile...) control the compiler, preventing it from reordering memory operations in the generated assembly code.
• Hardware barriers (dmb, mfence) control the CPU, preventing it from executing memory operations out-of-order at runtime.
• Correct lock-free synchronization requires a combination of these primitives to guard against both compiler and CPU reordering.

Preview of the Next Lesson:

We've now covered the most challenging aspects of kernel concurrency. Next, we will shift our focus within the kernel to another fundamental topic: memory management. We will start by exploring the two main functions for allocating kernel memory, kmalloc and vmalloc, and learn why you would choose one over the other. This is another classic interview topic that tests your understanding of virtual vs. physical memory within the kernel.