Implementing Python's Iterator Protocol

The Iteration Protocol: Iterable vs. Iterator

At its heart, Python's iteration is a simple contract between two types of objects:

1. Iterable: An object that can be looped over. It signals this ability by having an __iter__ dunder method. Its one job is to produce an iterator. Examples include lists, tuples, and strings.
2. Iterator: An object that manages the state of an iteration. It produces the next value with a __next__ method and signals the end by raising a StopIteration exception. It also has an __iter__ method that simply returns self.

When you write a for loop, Python performs these steps under the hood:

1. It calls the built-in iter() function on the iterable object (e.g., your list).
2. The iter() function calls the object's __iter__ method, which returns an iterator object.
3. The for loop then repeatedly calls the next() function on this iterator object to get each item.
4. When the iterator runs out of items, its __next__ method raises StopIteration, and the loop terminates.

This first video provides a foundational explanation of this distinction.

The Pitfall: Shared Iteration State

Now for the critical part of our learning outcome. Why is it a common mistake for an iterable container's __iter__ method to return self? Because it creates shared state. If __iter__ returns the same iterator instance every time, you can't have multiple, independent loops over the same object.

Consider how a list behaves. You expect to be able to do this:

my_list = ['a', 'b', 'c']




# Get two independent iterators from the same list
iter1 = iter(my_list)
iter2 = iter(my_list)

print(f"iter1 is iter2? {iter1 is iter2}") # False! They are different objects.




# Advancing one does not affect the other
print(f"Next from iter1: {next(iter1)}") # 'a'
print(f"Next from iter2: {next(iter2)}") # 'a'
print(f"Next from iter1: {next(iter1)}") # 'b'

If list.__iter__ simply returned self, both iter1 and iter2 would point to the same object. Advancing iter1 would also advance iter2, leading to confusing bugs.

This article section explains this concept perfectly.

The Correct Pattern: Separating the Iterable from the Iterator

To avoid shared state, the standard pattern is to use two classes:

1. The main container class (the iterable), which holds the data.
2. A private helper iterator class, which manages the state for a single iteration.

Let's design a WordList class that holds a sentence and allows iteration over its words.

Following this pattern, our WordList implementation would look like this:

# The Iterator class
class _WordListIterator:
    def __init__(self, words):
        self._words = words
        self._index = 0

    def __next__(self):
        if self._index >= len(self._words):
            raise StopIteration
        



        # Get the word and advance the index for the next call
        word = self._words[self._index]
        self._index += 1
        return word
    
    def __iter__(self):



        # An iterator's __iter__ should always return itself
        return self




# The Iterable class
class WordList:
    def __init__(self, text):
        self._words = text.split()

    def __iter__(self):



        # The key: always return a NEW, fresh iterator instance
        return _WordListIterator(self._words)




# --- Usage ---
sentence = WordList("Python's data model is powerful")




# The `for` loop automatically calls __iter__ to get a new iterator
for word in sentence:
    print(word)




# You can create multiple independent iterators
iter_a = iter(sentence)
iter_b = iter(sentence)

print(next(iter_a)) # Prints 'Python's'
print(next(iter_b)) # Also prints 'Python's'
print(next(iter_a)) # Prints 'data'

This separation guarantees that each for loop or call to iter(sentence) gets a clean start, with no risk of interference.

When Is It Okay for __iter__ to Return self?

So, is it always wrong for __iter__ to return self? No. It's perfectly correct for an object that is designed to be its own iterator.

These are typically objects that represent a single, non-resettable stream of data, not a reusable container. Think of a file handle or a network socket. You read from it once. You don't expect to create multiple independent "readers" from the same file handle object.

This video demonstrates creating a MyRange class that acts as its own iterator. This is a valid design because a range object is often used for a single pass in a loop.

The rule of thumb is:

• If your object is a reusable container (like a list, set, or our WordList), its __iter__ method must return a new iterator instance.
• If your object is an iterator itself (representing a single stream), its __iter__ method should return self.

Exercise: Make a Sentence Class Iterable

Your task is to properly implement the iteration protocol for the Sentence class below. You must use the two-class pattern (a Sentence iterable and a _SentenceIterator iterator) to ensure that multiple iterations are independent.

# sentence.py

class _SentenceIterator:
    """The stateful iterator for the Sentence class."""
    def __init__(self, words):



        # TODO: Initialize state here
        pass

    def __next__(self):



        # TODO: Return the next word or raise StopIteration
        pass

    def __iter__(self):



        # TODO: An iterator is its own iterable
        pass

class Sentence:
    """A container class that holds a sentence and is iterable."""
    def __init__(self, text: str):
        if not isinstance(text, str):
            raise TypeError("Sentence must be initialized with a string.")
        self.words = text.split()

    def __iter__(self):



        # TODO: Return a new, independent iterator object
        pass




# --- Tests to verify your implementation ---
my_sentence = Sentence("the quick brown fox")




# Test 1: Basic iteration
words = [word for word in my_sentence]
assert words == ["the", "quick", "brown", "fox"]




# Test 2: Ensure StopIteration is raised correctly
it = iter(my_sentence)
assert next(it) == "the"
assert next(it) == "quick"
assert next(it) == "brown"
assert next(it) == "fox"
try:
    next(it)



    # This line should not be reached
    assert False, "StopIteration was not raised"
except StopIteration:



    # This is the expected outcome
    pass




# Test 3: The most important test! Ensure iterators are independent.
iter1 = iter(my_sentence)
iter2 = iter(my_sentence)

assert iter1 is not iter2, "iter() should return a new iterator each time"
assert next(iter1) == "the"
assert next(iter1) == "quick"
assert next(iter2) == "the", "Iterator 2 should be independent of Iterator 1"

print("All tests passed!")

# sentence.py (solution)

class _SentenceIterator:
    """The stateful iterator for the Sentence class."""
    def __init__(self, words):
        self.words = words
        self.index = 0

    def __next__(self):
        if self.index >= len(self.words):
            raise StopIteration
        
        current_word = self.words[self.index]
        self.index += 1
        return current_word

    def __iter__(self):
        return self

class Sentence:
    """A container class that holds a sentence and is iterable."""
    def __init__(self, text: str):
        if not isinstance(text, str):
            raise TypeError("Sentence must be initialized with a string.")
        self.words = text.split()

    def __iter__(self):



        # The key to correctness: return a NEW instance of the iterator.
        return _SentenceIterator(self.words)

Conclusion

You have now implemented one of the most fundamental protocols in Python. By correctly separating the responsibilities of the iterable and the iterator, you can create custom container objects that are robust, predictable, and integrate seamlessly with the rest of the language.

Key Takeaways:

• An iterable has __iter__ and its job is to produce an iterator.
• An iterator has __next__ to provide values and __iter__ to return itself. It is stateful.
• The for loop uses this protocol by first getting an iterator, then repeatedly calling next() on it.
• For reusable containers, __iter__ must return a new, independent iterator instance to prevent bugs from shared state.
• Objects that are inherently single-pass streams (like file handles) can be their own iterators, in which case __iter__ correctly returns self.

Writing a separate iterator class can feel verbose. As you've seen in some of the resources, Python offers a much more elegant and concise way to create iterators.

In our next lesson, we'll explore generators and the yield keyword. You'll learn how to write a single __iter__ method that does the work of an entire iterator class in just a few lines of code.