Interning: Identity vs. Equality for Small Integers & Strings

Identity vs. Equality: The Foundation

Before we can discuss interning, we must be absolutely clear on the difference between Python's two equality-checking mechanisms: the is keyword and the == operator.

• The == operator checks for value equality. When you write a == b, you are asking, "Do these two objects have the same value?" Python answers this by calling the __eq__ method on the object a (i.e., a.__eq__(b)).
• The is keyword checks for object identity. When you write a is b, you are asking, "Are these two variables referencing the exact same object in memory?" This is equivalent to comparing their memory addresses using id(a) == id(b).

For mutable containers, this distinction is straightforward:

list_a = [1, 2, 3]
list_b = [1, 2, 3]

print(list_a == list_b)  # True, their contents are the same
print(list_a is list_b)  # False, they are two separate objects in memory

However, for immutable types like integers and strings, the behavior can be surprising due to interning. The following video provides a clear, animated explanation of this difference.

What is Object Interning and Why Does It Matter?

Object interning is a memory optimization technique where Python stores only one copy of certain immutable objects. When you create a new object that matches an already interned one, Python simply returns a reference to the existing object instead of creating a new one.

This has two major benefits that are critical in production systems:

1. Memory Savings: If your application uses thousands of instances of the string "user_id" or the integer 1, interning ensures only one object for each is ever stored in memory.
2. Performance Boost: Comparing interned objects using is is significantly faster than using ==. An identity check is a simple pointer comparison (a single machine instruction), whereas a value check for a long string requires iterating through and comparing every character. This is especially impactful for dictionary key lookups, where Python can often use a fast identity check as a shortcut before resorting to a full equality check.

The video below explains this performance implication clearly.

The image below illustrates this concept. On the left, with interning, multiple variables point to a single "Hello" object. On the right, without it, each variable gets its own separate copy, consuming more memory.

Integer Interning: The -5 to 256 Cache

CPython, the reference implementation of Python, takes a very specific approach to interning integers. At startup, it pre-allocates an array of integer objects for all integers in the inclusive range of -5 to 256. These are effectively singletons; any time you use an integer in this range, you are getting a reference to one of these pre-existing objects.

Let's see this in action:

# Integers within the cached range [-5, 256]
a = 256
b = 256
print(f"a is b for 256: {a is b}")  # True

c = -5
d = -5
print(f"c is d for -5: {c is d}")    # True




# Integer outside the cached range
x = 257
y = 257
print(f"x is y for 257: {x is y}")  # False

Why is x is y false? Because 257 is outside the pre-allocated cache, Python creates two distinct integer objects. They have the same value (x == y is True), but they are not the same object.

The following article provides an excellent deep dive into this CPython-specific behavior.

String Interning: Identifiers and the sys Module

String interning is more nuanced. Python doesn't intern all strings, as that could consume a large amount of memory for arbitrary text. Instead, CPython automatically interns string literals that follow certain rules.

The primary rule is that strings resembling Python identifiers (i.e., containing only letters, numbers, and underscores) are automatically interned. This is an optimization because such strings are frequently used as attribute names and dictionary keys.

# This looks like an identifier, so it gets interned
a = "hello_world"
b = "hello_world"
print(f"'hello_world' is interned: {a is b}")  # True




# This contains a space, so it's not an identifier
c = "hello world"
d = "hello world"
print(f"'hello world' is not interned: {c is d}") # False

For cases where you need to guarantee that a string is interned (perhaps for performance-critical dictionary lookups), Python provides an explicit mechanism: sys.intern().

import sys

c = "hello world"
d = "hello world"
print(f"Before interning: {c is d}")  # False

c_interned = sys.intern(c)
d_interned = sys.intern(d)
print(f"After interning: {c_interned is d_interned}") # True

The following video provides an exceptional journey into the CPython source code to reveal how string interning is implemented. This is the kind of deep knowledge that separates a user of Python from an expert.

The Golden Rule: is vs. == in Practice

Object interning is a CPython implementation detail. While it's fascinating and useful to understand, you should never write code that relies on implicit interning for correctness. Other Python implementations like PyPy or Jython may have different rules or may not intern at all.

This leads to the golden rule for production code:

• Use == to compare values. This is always the correct choice when you want to know if two objects represent the same thing.
• Use is only for identity checks against known singletons (like None) or when you have explicitly interned an object for performance reasons. The canonical example is if my_var is None:.

Relying on a is b for a = 257, b = 257 is a bug waiting to happen, as it may work in one context (due to the peephole optimizer) but fail in another.

Conclusion

In this lesson, we've pulled back another layer of Python's memory model to understand object interning. This optimization is a testament to Python's design philosophy: providing a high-level, easy-to-use language that is nonetheless highly optimized under the hood.

Here are the key takeaways:

• Interning is a memory and performance optimization where Python reuses existing immutable objects instead of creating new ones.
• The is operator checks for identity (same object in memory), while == checks for value (same content).
• CPython interns integers in the range [-5, 256]. Outside this range, identity is not guaranteed.
• CPython automatically interns string literals that look like identifiers (alphanumeric characters and underscores).
• You can force string interning with sys.intern() for performance-critical code that relies on fast identity checks.
• The Golden Rule: Always use == for value comparison. Only use is for checking against true singletons like None or for deliberate performance optimizations with explicitly interned objects.

We have now examined how Python manages object lifecycles, how it creates copies, and how it avoids creating objects. In our next and final lesson of this module, we will learn how to measure and analyze the memory usage of our Python objects and applications using tools like sys.getsizeof and tracemalloc, giving you the power to diagnose and optimize the memory footprint of your programs.