Python Generators: Creating Iterators with yield

In previous articles, you learned to create custom iterators by building a class implementing __iter__() and __next__(). While this approach is powerful, it is also verbose. Python provides a much more elegant and concise syntax: generator functions. A generator is a function that uses the yield keyword to produce a sequence of values lazily—one at a time, without storing the entire sequence in memory. This approach is ideal for processing large files, working with infinite sequences, and building data pipelines where memory efficiency matters.

Key Takeaways

• A generator is a simpler, more Pythonic way to create an iterator without manually implementing __iter__() and __next__().
• The yield keyword pauses a function and returns a value, saving the function's state so execution resumes from that point on the next call.
• Generators use lazy evaluation: values are computed on demand, not all at once. Processing a 10 GB file takes seconds; converting it to a list would exhaust memory.
• Any function containing yield is a generator function; calling it returns a generator object, not the result of executing the function.
• Generators are memory-efficient because they produce one item at a time, making them ideal for streaming, pagination, and large-scale data processing.

What Is a Generator and How Does It Differ from a Regular Function?

A generator is a function that yields values instead of returning a single value. The key difference lies in execution model:

• Regular function: Executes completely and returns a single value (or None). The function's local state is destroyed after return.
• Generator function: Pauses at each yield, returns a value, and saves its state. On the next call, execution resumes right after the yield, with all local variables intact.

# Regular function: returns once, state is destroyed
def regular_function():
    print("Starting")
    return 42
    print("This never runs")

result = regular_function()  # Output: Starting
print(result)  # Output: 42

# Generator function: yields multiple times, state is preserved
def generator_function():
    print("Starting")
    yield 1
    print("Resumed after first yield")
    yield 2
    print("Resumed after second yield")
    yield 3
    print("Done")

gen = generator_function()  # Output: nothing yet (function not executed)
print(next(gen))  # Output: Starting, then 1
print(next(gen))  # Output: Resumed after first yield, then 2
print(next(gen))  # Output: Resumed after second yield, then 3
print(next(gen))  # Output: Done, then StopIteration exception

When you call a generator function, it does not execute the code. Instead, it immediately returns a generator object—a special iterator. The generator object keeps track of the function's state, current position, and local variables. Each call to next(gen) resumes execution until the next yield statement.

Understanding the yield Keyword

The yield keyword is the heart of generators. It looks like return, but its behavior is fundamentally different:

• return: Exits the function permanently. The function's local variables are destroyed. The next call to the function starts from the beginning.
• yield: Pauses the function and sends a value to the caller. The function's local state is saved (including all local variables and the execution position). The next call to next(gen) resumes execution immediately after the yield.

def demo_yield():
    """Demonstrate yield behavior."""
    x = 10
    print("Before first yield")
    yield x
    
    x = 20
    print("Between yields")
    yield x
    
    x = 30
    print("After second yield")
    yield x

gen = demo_yield()

print("--- First next ---")
print(next(gen))  # Output: Before first yield, 10

print("\n--- Second next ---")
print(next(gen))  # Output: Between yields, 20

print("\n--- Third next ---")
print(next(gen))  # Output: After second yield, 30

print("\n--- Fourth next ---")
try:
    next(gen)
except StopIteration:
    print("Generator exhausted; StopIteration raised")

Notice that the variable x is modified and its value is preserved between yields. This is the core power of generators: they maintain state across pauses.

Your First Generator Function: Counting from Start to End

Let us rewrite a class-based iterator as a generator. This demonstrates the conciseness gain:

The old way (class-based iterator):

class Counter:
    """Custom iterator class; 10 lines of boilerplate."""
    def __init__(self, start, end):
        self.current = start
        self.end = end
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.current >= self.end:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

# Usage
counter = Counter(5, 8)
for num in counter:
    print(num)  # Output: 5, 6, 7

The new way (generator function):

def counter_generator(start, end):
    """Generator version; 5 lines of logic."""
    current = start
    while current < end:
        yield current
        current += 1

# Usage
for num in counter_generator(5, 8):
    print(num)  # Output: 5, 6, 7

The generator version is shorter, more readable, and requires no knowledge of __iter__() and __next__(). The while loop, the current variable, and incrementing logic are handled naturally. Python automatically implements the iterator protocol for you.

Tracing Execution with Print Statements

def counter_generator(start, end):
    """A generator with tracing."""
    print("Generator started...")
    current = start
    while current < end:
        print(f"Yielding {current}")
        yield current
        current += 1
    print("Generator finished.")

gen = counter_generator(5, 8)

print("\nAbout to call next for the first time...")
print(f"Received: {next(gen)}")
# Output: Generator started..., Yielding 5, Received: 5

print("\nAbout to call next for the second time...")
print(f"Received: {next(gen)}")
# Output: Yielding 6, Received: 6

print("\nNow, let's use it in a for loop...")
for num in gen:
    print(f"For loop received: {num}")
# Output: Yielding 7, For loop received: 7, Generator finished.

The tracing shows that:

1. The generator does not start until the first next() call.
2. Execution pauses at each yield and resumes at the next next() call.
3. The for loop uses the generator protocol automatically.

Why Use Generators? Readability and Memory Efficiency

Generators offer two major advantages over lists and class-based iterators:

1. Readability and Simplicity

Generator functions are intuitive. You write linear code with loops and conditionals, not class boilerplate. The logic reads naturally, like an algorithm description. For most iterator use cases, a generator is the idiomatic Python choice.

2. Memory Efficiency (The Critical Advantage)

Generators use lazy evaluation—they compute and yield values one at a time, without storing the entire sequence in memory. This is transformative for large datasets.

Bad approach (loading entire list into memory):

def read_log_file_bad(filepath):
    """Load entire file into memory; dangerous for large files."""
    with open(filepath, 'r') as f:
        return f.readlines()  # Returns a list of all lines

# Using this function:
lines = read_log_file_bad('massive_10gb.log')  # Allocates 10 GB of RAM
for line in lines:
    if "ERROR" in line:
        print(line)

If the file is 10 GB, this approach allocates 10 GB of memory for the list. For a billion-line file, you run out of RAM before processing a single line.

Good approach (generator yields one line at a time):

def read_log_file_good(filepath):
    """Yield one line at a time; constant memory usage."""
    with open(filepath, 'r') as f:
        for line in f:
            yield line.strip()  # Yield one line, keep minimal buffer

# Using this function:
for line in read_log_file_good('massive_10gb.log'):  # Uses < 1 MB RAM
    if "ERROR" in line:
        print(line)

Now memory usage is constant—only the current line is in memory. You can process terabyte-scale files on a laptop.

Performance Comparison

import sys

# List approach
def numbers_as_list(n):
    """Return a list of n numbers."""
    return list(range(n))

# Generator approach
def numbers_as_generator(n):
    """Yield n numbers one at a time."""
    for i in range(n):
        yield i

# Memory usage comparison
n = 10_000_000

# List: allocates memory for all 10 million integers
nums_list = numbers_as_list(n)
print(f"List size: {sys.getsizeof(nums_list)} bytes")  # Output: ~87 MB

# Generator: allocates only the generator object
nums_gen = numbers_as_generator(n)
print(f"Generator size: {sys.getsizeof(nums_gen)} bytes")  # Output: ~128 bytes

# Processing time is nearly identical, but memory is drastically lower
for num in nums_gen:
    if num == 5_000_000:
        print(f"Found: {num}")
        break

The generator object itself is tiny—it stores only the parameters and execution state. Values are computed on demand.

Frequently Asked Questions

Can I reset a generator to start from the beginning?

No, generators are one-time iterators. Once exhausted (all values yielded and function returns), you cannot reset them. To iterate again, create a new generator object by calling the generator function again. If you need to iterate multiple times over the same data, either use a list (if memory permits) or a generator function that returns a new generator each time.

def number_gen():
    yield 1
    yield 2
    yield 3

gen = number_gen()
for num in gen:
    print(num)  # Output: 1, 2, 3

for num in gen:
    print(num)  # Output: nothing (generator is exhausted)

# Create a new generator to iterate again
gen2 = number_gen()
for num in gen2:
    print(num)  # Output: 1, 2, 3

What is the difference between a generator and a list comprehension?

A list comprehension [x for x in range(10)] creates a list with all values in memory. A generator expression (x for x in range(10)) (note parentheses instead of brackets) is a generator that yields values lazily. For small datasets, the difference is negligible. For large datasets, generators are dramatically more memory-efficient. Use generator expressions for filtering and transforming large streams; use list comprehensions when you need the full list available.

# List comprehension: allocates full list
squares_list = [x**2 for x in range(1_000_000)]

# Generator expression: yields on demand
squares_gen = (x**2 for x in range(1_000_000))

# Both produce the same results, but squares_gen uses negligible memory
for sq in squares_gen:
    if sq > 1_000_000:
        break

Can generators call other generators?

Yes, and this is useful for composing complex data pipelines. A generator can yield from another generator, delegating to it.

def gen_a():
    yield 1
    yield 2

def gen_b():
    yield 3
    yield 4

def combined_gen():
    yield from gen_a()
    yield from gen_b()

for num in combined_gen():
    print(num)  # Output: 1, 2, 3, 4

The yield from statement automatically handles the iteration and exhaustion of the delegated generator.

What happens if I don't call next() on a generator; does it waste CPU?

No, a generator does not execute any code until you call next() on it. It is purely lazy. If you create a generator but never iterate, it consumes only the tiny generator object in memory. The function body only executes when you begin iteration.

How do I handle exceptions inside a generator?

Use try...except inside the generator function, just as you would in any function. If an exception occurs inside the try block, the except block handles it. If an unhandled exception occurs, iteration stops and the exception is raised to the caller.

def safe_divide(numerators, denominator):
    """Divide each numerator by the denominator; yield results."""
    for num in numerators:
        try:
            yield num / denominator
        except ZeroDivisionError:
            yield None  # Yield None for zero division

for result in safe_divide([10, 20, 30], 2):
    print(result)  # Output: 5.0, 10.0, 15.0

Further Reading

• Python Official Documentation: Generators
• Real Python: Generators and yield in Python
• Python Docs: yield Statement
• Real Python: Understanding Generators in Python