Understanding Undefined Behavior in C++

What is Undefined Behavior?

At its core, undefined behavior is the result of violating a rule of the C++ language for which the standard specifies no required outcome. This doesn't just mean your program might crash. It means the entire program, from that point onward—and sometimes even before—has no meaning.

The "demons flying out of your nose" analogy is famous but not very helpful. The reality is often more subtle and, for a developer, far more frightening. To get a precise sense of the real danger, let's start with a segment from Fedor Pikus's CppCon 2023 talk, "Undefined Behavior in C++: What Every Programmer Should Know and Fear."

It's also useful to distinguish UB from two other categories of behavior that are not fully specified:

• Unspecified behavior: The behavior may vary between compliant implementations, but the implementation is not required to document it. A classic example is the order of evaluation of function arguments. The compiler can choose any order, but it must choose one.
• Implementation-defined behavior: Similar to unspecified, but the implementation must document its choice. An example is the size of int. It will be a specific size on your platform/compiler, and this is documented.

Undefined behavior is the most dangerous of the three because it is a license for the compiler to do anything. Let's quickly see Fedor Pikus clarify these categories.

Why UB Enables "Time Travel" and Other Mysteries

The bizarre consequences of UB stem from a single, powerful assumption made by compilers: A correct C++ program does not contain undefined behavior. Based on this, the compiler performs the following logic:

1. Assume UB does not happen.
2. Deduce constraints on the program's state that must be true for UB to be avoided.
3. Optimize the code by assuming these constraints are always met.

If UB does occur at runtime, the initial assumption was false, but the standard doesn't care. The compiler's transformations are valid because the program's behavior was already undefined.

This logic is why UB can seem to "travel back in time." A potential UB-triggering operation later in a function can cause the compiler to remove a safety check at the beginning of the function.

A Tour of Common Undefined Behaviors

Let's ground this theory in the concrete examples you are likely to encounter in code and in interviews. The following article from Project Nayuki provides an excellent catalogue of common UB triggers.

Now, let's dive deeper into a few of the most important categories.

1. Signed Integer Overflow

While unsigned integer overflow is well-defined (it wraps around), signed integer overflow is undefined behavior. This is perhaps the most famous example of a UB-based optimization.

Consider the check x + 1 > x. Mathematically, this is always true. In two's complement arithmetic, it's false when x is INT_MAX. But in C++, the compiler assumes a correct program will never let x + 1 overflow. Therefore, it can optimize the check x + 1 > x to simply true.

This assumption allows the compiler to eliminate checks it deems impossible. Look at the C++ code on the left and the assembly generated by the Clang compiler on the right.

2. Null Pointer Dereference

This is the quintessential "time-traveling" bug. You might think dereferencing a null pointer simply crashes at the point of dereference. The reality is far more sinister. If a compiler sees a line *p;, it can assume p is never null anywhere in that code path. This can lead it to remove null checks that occur before the program would have crashed!

This next video segment is arguably the most important one in this lesson. It demonstrates how a null-check guard can be optimized away, causing a crash on a line that looks perfectly safe.

3. Out-of-Bounds Memory Access

Accessing an array or buffer outside its allocated bounds is UB. This can corrupt adjacent memory, leading to unpredictable behavior. On a typical process memory layout, an out-of-bounds write on the stack (a "buffer overflow") could overwrite other local variables, the stored frame pointer, or critically, the function's return address.

4. Strict Aliasing Violations

This is a more subtle but crucial topic for high-performance C++. The strict aliasing rule states that two pointers of different types cannot point to the same memory location (with some exceptions). This gives the compiler a powerful optimization tool: if it sees a write through a float* and a read from an int*, it can assume they don't alias and can reorder the operations freely without re-reading the memory location. If you violate this rule (e.g., by casting a pointer), you invoke UB.

John Regehr provides a very clear example in his CppCon 2017 talk.

Defending Against Undefined Behavior

You cannot test for undefined behavior in the traditional sense, because a program with UB is not a valid program. Defense must be proactive.

1. Know the Rules: The first line of defense is awareness. Knowing what constitutes UB (integer overflow, using dangling pointers, etc.) is the first step to avoiding it.

2. Enable All Warnings: Compile your code with the highest warning levels (e.g., g++ -Wall -Wextra -Wpedantic or /W4 in MSVC). Compilers are getting better at spotting potential UB at compile time.

3. Use Sanitizers: Modern compilers provide powerful dynamic analysis tools called sanitizers. You compile your code with a special flag, and the resulting executable will detect and report UB at runtime when your test suite runs.

   • AddressSanitizer (ASan): Detects memory errors like out-of-bounds access and use-after-free. (-fsanitize=address)
   • UndefinedBehaviorSanitizer (UBSan): Detects UB like signed integer overflow, illegal shifts, and null pointer dereferences. (-fsanitize=undefined)

Let's hear Fedor Pikus give a quick overview of UBSan.

4. Vigilant Code Reviews: As a senior engineer and tech lead, promoting a culture of UB-awareness during code reviews is vital. If you see a potentially unsafe operation, it is your responsibility to ask for the "proof" that its preconditions are met.

Conclusion

This lesson completes our first module on C++ fundamentals. We've moved from the well-defined world of exceptions to the treacherous territory of undefined behavior.

Here are the key takeaways:

• Undefined Behavior is a Contract Violation: It occurs when your code violates a language rule, rendering the entire program's behavior meaningless from the C++ standard's perspective.
• Optimization is the Motive: UB exists to give compilers the freedom to generate highly optimized code by assuming your program is always correct.
• Consequences are Counter-Intuitive: UB can cause "time travel" where code is optimized away based on operations that happen later, leading to crashes in seemingly safe code.
• Common Triggers: Be vigilant about signed integer overflow, null pointer dereferencing, out-of-bounds access, use of dangling pointers/uninitialized variables, and strict aliasing violations.
• Proactive Defense is Key: Use high compiler warning levels, sanitizers (ASan, UBSan), and rigorous code reviews to catch UB before it causes problems.

You now have a solid foundation in the principles of memory management, object lifecycle, exception safety, and the pitfalls of undefined behavior. This knowledge is the bedrock upon which robust C++ applications are built.

In our next module, we will shift gears and begin our exploration of Modern C++. We will start by examining the core concepts that enable move semantics—lvalues and rvalues—and see how they allow for radical new levels of efficiency in resource management.