Generator Expressions: Memory-Efficient Lazy Evaluation
Generator expressions are the concise, memory-efficient alternative to list comprehensions. They use parentheses instead of square brackets and create generator objects that compute values on-demand (lazy evaluation) rather than building entire lists in memory. For large datasets, files, or streaming operations, generator expressions drastically reduce memory usage while maintaining readable syntax.
Key Takeaways
- Generator expressions use
(expr for item in iterable)syntax (parentheses, not brackets) to create generators - They compute values lazily—one item at a time—rather than building the full list upfront
- Memory usage is constant and independent of dataset size; list comprehensions allocate all items simultaneously
- Ideal for large datasets, file processing, and feeding data to functions like
sum(),max(),min()
Generator Expressions vs. List Comprehensions: The Syntax Difference
The only syntactic difference between a list comprehension and generator expression is the bracket type:
# List comprehension: uses [ ]
squares_list = [x * x for x in range(10)]
print(squares_list) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
print(type(squares_list)) # <class 'list'>
# Generator expression: uses ( )
squares_gen = (x * x for x in range(10))
print(squares_gen) # <generator object <genexpr> at 0x...>
print(type(squares_gen)) # <class 'generator'>
List comprehension evaluates immediately and returns a full list with all 10 values in memory. Generator expression creates a generator object that computes values only when requested.
Iterating Over Each Type
# List: all values are already computed
squares_list = [x * x for x in range(5)]
for num in squares_list:
print(num, end=" ")
# Output: 0 1 4 9 16
# Generator: values computed on each iteration
squares_gen = (x * x for x in range(5))
for num in squares_gen:
print(num, end=" ")
# Output: 0 1 4 9 16 (same result, but computed on-the-fly)
Once a generator is exhausted (iterated over completely), it cannot be reused:
gen = (x * 2 for x in range(3))
print(list(gen)) # [0, 2, 4]
print(list(gen)) # [] (generator is exhausted)
Memory Efficiency: The Real Power of Generators
The memory advantage becomes dramatic with large datasets.
Memory Comparison: List vs. Generator
import sys
# List comprehension: stores all 10 million items
list_comp = [i * i for i in range(10_000_000)]
print(f"List memory: {sys.getsizeof(list_comp):,} bytes") # ~80-90 MB
# Generator expression: stores generator object only (~100-200 bytes)
gen_exp = (i * i for i in range(10_000_000))
print(f"Generator memory: {sys.getsizeof(gen_exp):,} bytes") # ~128 bytes
The list allocates ~80 MB. The generator allocates ~128 bytes regardless of size.
Streaming Large Files
Imagine a 5 GB log file. With a list comprehension, you'd load the entire file into RAM (crash). With a generator, you process line-by-line:
# List comprehension: loads entire file into memory
lines_list = [line.strip() for line in open('large_file.txt')] # 5 GB in RAM!
# Generator expression: processes one line at a time
lines_gen = (line.strip() for line in open('large_file.txt'))
for line in lines_gen:
process(line) # Each line is read, processed, then discarded
The generator uses constant memory regardless of file size.
Generator Expression Syntax: Filtering and Transformation
Generator expressions support all the power of list comprehensions, including conditional filtering:
# Simple generator
squares = (x * x for x in range(10))
# With filtering
even_squares = (x * x for x in range(10) if x % 2 == 0)
print(list(even_squares)) # [0, 4, 16, 36, 64]
# Nested iteration
pairs = ((x, y) for x in range(3) for y in range(3))
print(list(pairs)) # [(0, 0), (0, 1), ..., (2, 2)]
# Complex transformation
lengths = (len(word) for word in ['apple', 'pie', 'cake'])
print(list(lengths)) # [5, 3, 4]
Using Generators as Function Arguments
When passing a generator as the sole argument to a function, the outer parentheses can be omitted:
# Standard: extra parentheses
total = sum((i for i in range(101) if i % 2 != 0))
# Cleaner: parentheses optional when generator is the only argument
total = sum(i for i in range(101) if i % 2 != 0)
print(f"Sum of odd numbers 1-100: {total}") # 2500
# Works with other functions too
largest = max(x * x for x in range(1, 6))
print(f"Largest square: {largest}") # 25
smallest = min(abs(x) for x in [-5, -2, 3, 0, 1])
print(f"Smallest absolute value: {smallest}") # 0
This is idiomatic Python and frequently used in real code.
Practical Applications of Generator Expressions
1. Processing Large Datasets
# Read and process a massive CSV file
data_file = open('sales_data.csv')
expensive_items = (
float(row.split(',')[2])
for row in data_file
if float(row.split(',')[2]) > 1000
)
total = sum(expensive_items)
print(f"Total expensive item sales: ${total:,.2f}")
2. Chaining Multiple Generators
# Create a pipeline: read → filter → transform → filter again
numbers = (int(line.strip()) for line in open('numbers.txt'))
positive = (n for n in numbers if n > 0)
doubled = (n * 2 for n in positive)
under_100 = (n for n in doubled if n < 100)
print(list(under_100)) # Lazy evaluation through entire chain
3. Generating Sequences On-Demand
# Generate Fibonacci numbers indefinitely (without storing list)
def fibonacci_gen():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
# Use generator expression to limit and filter
first_10_evens = (
n for n in fibonacci_gen()
if n % 2 == 0
for _ in range(10) # Limit iterations
)
(Note: This requires breaking out of the infinite loop; typically use itertools.islice().)
4. Memory-Efficient Counting
# Count items matching a condition without storing them
log_file = open('app.log')
error_count = sum(1 for line in log_file if 'ERROR' in line)
print(f"Total errors: {error_count}")
Generator Expressions vs. Generator Functions
Generator functions use def and yield:
def square_gen(n):
for i in range(n):
yield i * i
gen = square_gen(5)
print(list(gen)) # [0, 1, 4, 9, 16]
Generator expressions are one-liners:
gen = (i * i for i in range(5))
print(list(gen)) # [0, 1, 4, 9, 16]
| Feature | Generator Function | Generator Expression |
|---|---|---|
| Syntax | def keyword + yield | Single-line comprehension syntax |
| Reusability | Callable multiple times | Single-use (exhausted after iteration) |
| Complexity | Good for complex logic | Best for simple transformations |
| Readability | Full function body | Compact, inline |
Use generator functions for complex logic; use generator expressions for simple transformations.
Performance Comparison: Real Numbers
import time
import sys
# Scenario: Sum of squares for 10 million numbers
# List comprehension
start = time.time()
list_result = sum([i * i for i in range(10_000_000)])
list_time = time.time() - start
list_memory = sys.getsizeof([i * i for i in range(10_000_000)])
# Generator expression
start = time.time()
gen_result = sum(i * i for i in range(10_000_000))
gen_time = time.time() - start
gen_memory = sys.getsizeof((i * i for i in range(10_000_000)))
print(f"Results: list={list_result}, gen={gen_result}")
print(f"Time: list={list_time:.3f}s, gen={gen_time:.3f}s")
print(f"Memory: list={list_memory:,}B, gen={gen_memory:,}B")
# Time is similar; memory difference is massive
Results: Execution time is comparable, but the generator uses 1,000× less memory.
When to Use Generator Expressions
Use generator expressions for:
- Large datasets or streams that don't fit in memory
- Processing files line-by-line
- Feeding data into functions (
sum(),max(),any(),all()) - Chaining multiple transformations
- When you iterate once and don't need to store results
Use list comprehensions for:
- Small, known-size datasets
- When you need random access to elements (indexing)
- When you need to iterate multiple times
- When the result is used immediately in multiple places
# Generator: Process once
for line in (l.strip() for l in open('file.txt')):
print(line)
# List: Reuse multiple times
lines = [l.strip() for l in open('file.txt')]
print(len(lines))
print(lines[0])
print(lines[-1])
Frequently Asked Questions
Can I convert a generator to a list?
Yes, use list():
gen = (x * 2 for x in range(5))
lst = list(gen)
print(lst) # [0, 2, 4, 6, 8]
print(list(gen)) # [] (generator exhausted)
Can I iterate a generator multiple times?
No. Once exhausted, it's done. Create a new generator if you need to iterate again:
gen1 = (x * 2 for x in range(3))
print(list(gen1)) # [0, 2, 4]
print(list(gen1)) # [] (exhausted)
gen2 = (x * 2 for x in range(3)) # New generator
print(list(gen2)) # [0, 2, 4]
Should I always use generators instead of lists?
No. Use generators for memory efficiency and lazy evaluation. Use lists when you need indexing, multiple iterations, or the data fits comfortably in memory.
Can generator expressions be nested?
Yes:
matrix = [[1, 2], [3, 4], [5, 6]]
flat = (num for row in matrix for num in row)
print(list(flat)) # [1, 2, 3, 4, 5, 6]
What's the difference between next() and iterating?
next() pulls the next value; iterating does this automatically:
gen = (x * 2 for x in range(3))
print(next(gen)) # 0
print(next(gen)) # 2
print(next(gen)) # 4
print(next(gen)) # StopIteration exception