Simplifying Iterators with Generators and `yield`

From Verbose Class to Concise Generator

Let's start by directly comparing the two approaches. In the last lesson, we saw how to build an iterator class that keeps track of its state (_index) and implements __next__. The video below first recaps that class-based approach and then immediately introduces the generator-based alternative, which is the core of today's lesson.

The difference is stark. The generator function is not just shorter; it's more expressive. The logic for producing the sequence is all in one place, unencumbered by the boilerplate of the iterator protocol.

The Magic of yield

So, what is happening? The yield keyword is the key. When you include yield in a function, it's no longer a regular function. It becomes a generator function.

Here's the mechanism:

1. Calling a generator function does not execute it. Instead, it returns a generator object.
2. This generator object is a fully-formed iterator. Python automatically implements the __iter__ and __next__ methods for you.
3. Each time next() is called on the generator object, the function's code executes until it hits a yield statement.
4. The yield statement does two things: it provides the value to the caller, and it pauses the function's execution, preserving its entire local state (variables, loops, etc.).
5. When next() is called again, the function resumes right after the yield statement where it left off.
6. When the function completes (either by reaching the end or a return statement), it automatically raises StopIteration.

This article explains the process clearly.

The Idiomatic Pattern: Using yield within __iter__

While you can use a standalone generator function, the most common and powerful pattern is to place the generator logic directly inside your container class's __iter__ method.

By doing this, the __iter__ method itself becomes a generator function. Every time it's called (e.g., by a for loop), it returns a new, independent generator object. This elegantly solves the shared state problem we discussed in the last lesson, without needing a separate iterator class.

Let's refactor the Sentence class from our previous exercise:

Before (Two-Class Pattern):

class _SentenceIterator:
    def __init__(self, words):
        self.words = words
        self.index = 0

    def __next__(self):
        if self.index >= len(self.words):
            raise StopIteration
        current_word = self.words[self.index]
        self.index += 1
        return current_word

    def __iter__(self):
        return self

class Sentence:
    def __init__(self, text: str):
        self.words = text.split()

    def __iter__(self):



        # Returns an instance of the helper class
        return _SentenceIterator(self.words)

After (Generator Pattern):

class Sentence:
    def __init__(self, text: str):
        self.words = text.split()

    def __iter__(self):



        # __iter__ is now a generator function!
        for word in self.words:
            yield word



        # No need for a separate iterator class.
        # No need for self.index state.
        # No need to manually raise StopIteration.

This is the Pythonic way. The _SentenceIterator class is now completely unnecessary. The __iter__ method concisely expresses the iteration logic.

This article from Trey Hunner demonstrates this exact pattern and declares generators as the way to make iterators.

Generator Expressions: The Comprehension-style Iterator

For very simple cases, you can create a generator without even a def statement using a generator expression. The syntax is identical to a list comprehension, but using parentheses () instead of square brackets [].

• List Comprehension (builds a list in memory): [x*x for x in range(10)]
• Generator Expression (creates a lazy iterator): (x*x for x in range(10))

Generator expressions are memory-efficient for simple transformations because they produce items one at a time, just like a generator function.

# Create a large list in memory
list_comp = [i for i in range(10_000_000)] 



# list_comp consumes significant memory




# Create a generator object (consumes almost no memory)
gen_exp = (i for i in range(10_000_000)) 




# You can iterate over it just like any other iterator
print(next(gen_exp)) # 0
print(next(gen_exp)) # 1



# ...and so on

We'll dive deeper into comprehensions in a later module, but it's important to recognize this syntax now as another way to create generators.

Summary: Custom Iterators vs. Generators

The choice between a manual iterator class and a generator is clear in most cases. This table summarizes the trade-offs.

In short: always prefer a generator unless you have a compelling reason not to. A generator is simpler to write, easier to read, and less prone to errors.

This diagram visualizes where generators fit within Python's iteration landscape.

Exercise: Refactor to Use yield

Your task is to refactor the following ReversedSequence class. It currently uses a verbose, stateful iterator class. Your goal is to eliminate the _ReversedSequenceIterator class and implement the __iter__ method as a generator function.

# original_code.py

class _ReversedSequenceIterator:
    """A stateful iterator that yields items from a sequence in reverse."""
    def __init__(self, sequence):
        self.sequence = sequence
        self.index = len(sequence)

    def __next__(self):
        if self.index == 0:
            raise StopIteration
        self.index -= 1
        return self.sequence[self.index]

    def __iter__(self):
        return self

class ReversedSequence:
    """A container that allows iterating over a sequence in reverse."""
    def __init__(self, sequence):
        self.sequence = sequence
    
    def __iter__(self):



        # TODO: This implementation is too verbose.
        # Replace it with a generator function using yield.
        return _ReversedSequenceIterator(self.sequence)




# --- Tests ---
data = [10, 20, 30, 40]
rev_seq = ReversedSequence(data)




# Test 1: Iteration works
assert list(rev_seq) == [40, 30, 20, 10]




# Test 2: Independent iteration
iter1 = iter(rev_seq)
iter2 = iter(rev_seq)
assert next(iter1) == 40
assert next(iter1) == 30
assert next(iter2) == 40, "Iterators should be independent"

print("All tests passed! (with the original code)")
print("Now, refactor it to be more Pythonic.")

Your goal is to modify ReversedSequence so the tests still pass, but the _ReversedSequenceIterator class is no longer needed.

# refactored_code.py

class ReversedSequence:
    """A container that allows iterating over a sequence in reverse."""
    def __init__(self, sequence):
        self.sequence = sequence
    
    def __iter__(self):
        """
        Implementation using a generator function.
        It's more concise and readable.
        """



        # A simple way to iterate backwards is to use reversed()
        # but for demonstration, we can use a while loop.
        index = len(self.sequence)
        while index > 0:
            index -= 1
            yield self.sequence[index]




# --- Tests ---
data = [10, 20, 30, 40]
rev_seq = ReversedSequence(data)




# Test 1: Iteration works
assert list(rev_seq) == [40, 30, 20, 10]




# Test 2: Independent iteration
iter1 = iter(rev_seq)
iter2 = iter(rev_seq)
assert next(iter1) == 40
assert next(iter1) == 30
assert next(iter2) == 40, "Iterators should be independent"

print("All tests passed with the refactored code!")

Conclusion

You've now mastered a cornerstone of idiomatic Python. By replacing manual iterator classes with generators, you write code that is not only shorter but also clearer and more maintainable.

Key Takeaways:

• A function containing yield is a generator function that produces a generator object (an iterator).
• yield pauses a function's execution and saves its state, resuming on the next call to next().
• The most Pythonic way to make a container iterable is to implement its __iter__ method as a generator function.
• This approach automatically handles state management and StopIteration, eliminating boilerplate.
• Generator expressions (item for item in iterable) offer a concise syntax for simple, lazy iteration.

In the next lesson, we will continue expanding our custom objects' capabilities. We'll implement the container protocols __len__ and __contains__, allowing the built-in len() function and the in operator to work seamlessly with your types.