Integer Overflow: Signed vs. Unsigned Behavior

Unsigned Integers: The Predictable Wraparound

Unsigned integers in C have a behavior that is easy to predict because it is well-defined by the standard. They act like an odometer in a car: after reaching its maximum value, it "wraps around" back to zero. The same happens in reverse for underflow; subtracting 1 from 0 causes it to wrap around to the maximum possible value.

This behavior occurs because of the underlying binary arithmetic. For example, an 8-bit unsigned integer can hold values from 0 to 255. In binary, 255 is 11111111. If you add 1, you get 100000000. But since the type can only store 8 bits, the leading 1 is discarded, leaving 00000000, which is 0.

Video games, especially older ones, provide some classic examples of this. The following video illustrates this wraparound behavior with bugs and mechanics from Pac-Man and Final Fantasy.

You can easily replicate this yourself with a short C program:

#include <stdio.h>

int main(void) {
    unsigned char u_char_val = 255;
    printf("Initial unsigned char: %u\n", u_char_val);
    
    u_char_val = u_char_val + 1;
    printf("After adding 1 (overflow): %u\n", u_char_val);

    u_char_val = u_char_val - 1;
    printf("After subtracting 1 (underflow): %u\n", u_char_val);
    
    return 0;
}

When you run this, you will see the overflow wrap to 0 and the subsequent underflow wrap back to 255. This predictable behavior is something you can, in rare cases, even use intentionally, but more often it's something you need to guard against.

A famous, non-malicious bug caused by unsigned integer underflow is the "Nuclear Gandhi" bug in the game Civilization. This next reading explains how a simple subtraction caused a peaceful leader to become the most aggressive character in the game.

Signed Integers: The Danger of Undefined Behavior

While unsigned overflow is predictable, signed integer overflow in C is Undefined Behavior (UB). This is one ofthe most critical concepts to understand for safe C programming.

"Undefined Behavior" means the C standard places no requirements on what must happen. When signed overflow occurs:

• The program might crash.
• The value might wrap around (e.g., from INT_MAX to INT_MIN), which is common on systems using two's complement representation, but you cannot rely on this.
• The compiler, assuming that signed overflow will never happen, might perform optimizations that lead to completely nonsensical results or security holes.

Because the behavior is not guaranteed, you must treat any potential for signed integer overflow as a critical bug and prevent it from ever happening.

The following article from ESET provides a great visual for how both signed and unsigned integers wrap around, but remember, only the unsigned case is guaranteed by the C standard.

Techniques for Handling and Preventing Overflow

Since signed overflow is UB and unsigned overflow is often undesirable, robust code must prevent it. There are several strategies to achieve this.

1. Pre-computation Checks

The most direct method is to check if an operation would cause an overflow before you perform it. For example, to check if a + b will overflow a signed integer:

#include <limits.h> // For INT_MAX

// a and b are positive integers
if (a > INT_MAX - b) {
    // Adding a and b would cause an overflow.
    // Handle the error here.
} else {
    int sum = a + b;
    // Proceed normally.
}

This logic checks if b is larger than the remaining "space" available in the integer after a has been accounted for. Similar checks exist for subtraction and multiplication.

2. Algorithmic Redesign

Sometimes, you can rewrite a formula to avoid the large intermediate values that cause overflow. This is often more elegant and efficient than manual checks. A classic example is finding the middle index in an array for a binary search.

The naive approach is middle = (left + right) / 2;. If left and right are both very large, their sum can overflow INT_MAX. The robust solution is mathematically equivalent but avoids the large sum: middle = left + (right - left) / 2;.

The following video provides an excellent, practical demonstration of this exact problem and its solution.

3. Saturation Arithmetic

Another approach, though less common in general C but prevalent in graphics and signal processing, is saturation. Instead of wrapping around, the value "clamps" or "saturates" at the maximum (or minimum) value. For example, with saturation, INT_MAX + 1 would simply remain INT_MAX. C does not do this by default, but you can implement it with explicit checks.

Why This All Matters: A Gateway to Security Flaws

Integer overflow isn't just a mathematical curiosity; it is a frequent source of serious security vulnerabilities. One of the most dangerous consequences is that an integer overflow can be used to trigger a buffer overflow.

Imagine a program that accepts a length from a user, allocates a buffer of that size, and copies data into it. A check like if (user_length > MAX_LENGTH) seems safe. But what if a malicious user provides a negative length? If the length variable is signed, the check might pass. However, when that negative number is then used in a function that expects an unsigned size (like malloc or memcpy), it can be implicitly converted, wrapping around to a massive positive number. This can cause the program to attempt a memory operation far beyond the intended buffer's bounds, leading to crashes or, worse, arbitrary code execution.

This exact scenario is detailed in the ESET article, providing a tangible example of how a simple integer bug becomes a critical security flaw.

Conclusion

In this lesson, you've moved from knowing what integers are to understanding how they behave under stress. This is a vital step in your journey toward writing professional, low-level C code. The distinction between the predictable wraparound of unsigned types and the treacherous Undefined Behavior of signed types is paramount.

Key Takeaways:

• Unsigned integer overflow/underflow is well-defined: The value wraps around, like a clock or an odometer.
• Signed integer overflow is Undefined Behavior: You must never allow it to happen, as the results are unpredictable and can be exploited by attackers.
• Prevention is the best strategy: You can handle potential overflows by using pre-computation checks to validate operations or by redesigning algorithms to avoid large intermediate values.
• Integer overflow is a security risk: It can be the root cause of more severe vulnerabilities like buffer overflows, making it critical to manage in any application.

We saw in the last example that an implicit, automatic type conversion was the final trigger for the security bug. The C compiler often performs these conversions behind the scenes, and they can be a source of very subtle errors. In our next lesson, we will shine a spotlight on these implicit type conversions and integer promotions to understand the rules the compiler follows and how to avoid the pitfalls they can create.