Skip to main content

Generator Expressions: A Memory-Efficient Way to Create Generators

In the last article, we saw how generator functions using yield provide a memory-efficient way to create iterators. Python offers an even more concise and elegant way to create simple generators: generator expressions.

A generator expression looks very similar to a list comprehension, but instead of building a full list in memory, it creates a generator object. This allows for the same powerful, memory-saving, lazy evaluation in a more compact syntax.

📚 Prerequisites​

You should understand list comprehensions and the basic concept of generators.

🎯 Article Outline: What You'll Master​

In this article, you will learn:

  • ✅ Generator Expression Syntax: How to create a generator using a syntax similar to a list comprehension.
  • ✅ The Key Difference: Understand why () creates a generator while [] creates a list.
  • ✅ Memory Efficiency: See a practical example of how generator expressions save memory compared to list comprehensions.
  • ✅ When to Use Them: Identify the ideal situations for using a generator expression.

🧠 Section 1: From List Comprehension to Generator Expression​

Let's start with a familiar list comprehension. It builds a new list by performing an operation on each item in an existing iterable.

List Comprehension:

# This creates a list, holding all 10 numbers in memory at once.
squares_list = [x * x for x in range(10)]
print(f"List Comprehension: {squares_list}")

Now, to turn this into a generator expression, we simply replace the square brackets [] with parentheses ().

Generator Expression:

# This creates a generator object. It holds no numbers in memory yet.
squares_generator = (x * x for x in range(10))
print(f"Generator Expression: {squares_generator}")

Output:

List Comprehension: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Generator Expression: <generator object <genexpr> at 0x10d1e3c10>

Notice the difference. The list comprehension immediately created the full list of 10 numbers. The generator expression, however, created a generator object. It hasn't calculated any squares yet; it's just waiting to be asked.

To get the values from the generator, you iterate over it, just like any other iterator:

for num in squares_generator:
    print(num, end=" ") # Output: 0 1 4 9 16 25 36 49 64 81

💻 Section 2: The Memory Advantage​

The real power of generator expressions becomes clear when working with large sequences.

Let's calculate the sum of the squares of the first 10 million numbers.

Using a List Comprehension (High Memory Usage):

import sys

# This will create a list with 10,000,000 numbers in memory first.
list_comp = [i * i for i in range(10_000_000)]
print(f"Memory used by list: {sys.getsizeof(list_comp)} bytes")

total_list = sum(list_comp)
print(f"Sum from list: {total_list}")

On a typical machine, this might use over 80 MB of RAM just to hold the list of squares before it can even be summed.

Using a Generator Expression (Low Memory Usage):

import sys

# This creates a tiny generator object. No list is ever built.
gen_exp = (i * i for i in range(10_000_000))
print(f"Memory used by generator: {sys.getsizeof(gen_exp)} bytes")

# The sum() function pulls one number at a time from the generator.
total_gen = sum(gen_exp)
print(f"Sum from generator: {total_gen}")

The generator object itself is tiny (usually around 100-200 bytes), regardless of the number of items it's set to produce. The sum() function pulls each squared number from the generator one by one, adds it to the running total, and then discards it. The memory usage is constant and extremely low.

This makes generator expressions the clear choice when working with large data streams, files, or any situation where you don't need to have all the results in memory at once.

🛠️ Section 3: Syntax and Use Cases​

Generator expressions share the same powerful syntax as list comprehensions, including if conditions.

# A generator of even numbers from 0 to 18
even_numbers = (n for n in range(20) if n % 2 == 0)

for num in even_numbers:
    print(num, end=" ") # Output: 0 2 4 6 8 10 12 14 16 18

A common and elegant use case is when passing a generator directly to another function. If the generator expression is the only argument to a function, you don't need the inner parentheses.

# The sum() function is a perfect consumer for a generator expression.
# Note the lack of extra parentheses around the generator.
total = sum(i for i in range(101) if i % 2 != 0) # Sum of odd numbers from 1 to 100
print(f"\nSum of odd numbers: {total}")

✨ Conclusion & Key Takeaways​

Generator expressions are a concise and memory-efficient tool that combines the readability of list comprehensions with the power of lazy evaluation from generators.

Let's summarize the key takeaways:

  • Syntax: Use parentheses () instead of square brackets [] to create a generator expression.
  • Lazy Evaluation: Generator expressions don't build a full list in memory. They create a generator object that produces values one at a time, on demand.
  • Memory Efficiency: They are vastly more memory-efficient than list comprehensions for large datasets.
  • Use Cases: They are ideal for calculations on large sequences, processing lines from large files, and as arguments to functions that consume iterables (like sum(), min(), max()).

Challenge Yourself: A "lazy" file reader can be created with a generator expression. If you have a text file my_file.txt, the expression (line.strip() for line in open('my_file.txt')) will create a generator that yields each line without reading the whole file into memory. Try using this pattern to find the longest line in a text file you create.

➡️ Next Steps​

We have now fully explored iterators and generators. The final topic in this advanced series is another powerful Python feature that modifies the behavior of functions: Decorators. In the next article, we'll dive into "Decorators (Part 1): Introduction to decorators."

Happy coding!