Skip to main content

Iterators and Iterables: Understanding the Iterator Protocol

The iterator protocol is one of the most elegant and powerful design patterns in Python, underpinning the for loop and enabling memory-efficient iteration over data of any size. Understanding how iterables and iterators work—the difference between them, the two magic methods __iter__() and __next__(), and how to create custom iterators—unlocks advanced Python features like generators and comprehensions. This guide demystifies the protocol with clear explanations and runnable examples.

Key Takeaways​

  • Iterable vs. Iterator: An iterable is any object with an __iter__() method (lists, tuples, strings); an iterator is the object that tracks position and implements __next__().
  • The Iterator Protocol: The for loop calls iter() on an iterable to get an iterator, then calls next() repeatedly until StopIteration is raised.
  • Lazy Evaluation: Iterators produce one value at a time only when requested, not upfront, making them memory-efficient for large datasets.
  • Custom Iterators: Implement __iter__() and __next__() in a class to create your own iterator for any custom sequence.
  • Memory Efficiency: An iterator can yield 1 billion values without storing all of them in memory, unlike a list which requires upfront allocation.

What Is the Iterator Protocol?​

The iterator protocol is a design pattern that lets Python's for loop work uniformly across different types of objects—lists, strings, dictionaries, files, and custom classes. It is defined by two methods:

  1. __iter__(self): Called on an iterable; returns an iterator object.
  2. __next__(self): Called on an iterator; returns the next value in the sequence. Raises StopIteration when there are no more values.

When you write for item in my_list:, Python silently performs these steps:

  1. Calls iter(my_list) → my_list.__iter__() to get an iterator.
  2. Calls next() on the iterator repeatedly.
  3. Catches StopIteration to exit the loop cleanly.

Understanding this protocol is foundational to Python mastery.

Iterable vs. Iterator: The Key Distinction​

These terms sound similar but have distinct, precise meanings.

What Is an Iterable?​

An iterable is any Python object that you can loop over with a for loop. Formally, it is an object with an __iter__() method that returns an iterator.

Examples of iterables:

  • Lists, tuples, sets
  • Strings
  • Dictionaries (iterate over keys by default)
  • Files
  • Ranges
  • Any custom class implementing __iter__()

Analogy: An iterable is like a book. It contains all the items (pages), but it isn't responsible for keeping track of your reading progress. Every time you start reading, you get a fresh iterator.

my_list = [10, 20, 30]
print(hasattr(my_list, '__iter__'))  # Output: True (it's an iterable)

What Is an Iterator?​

An iterator is the object that actually performs the iterating—it tracks the current position in the sequence and produces the next value on demand. An iterator is defined by having a __next__() method (and technically also __iter__(), which returns itself).

Analogy: An iterator is like a bookmark. It knows exactly where you are in the book (the iterable) and can provide the next page when you ask.

my_list = [10, 20, 30]
my_iterator = iter(my_list)
print(hasattr(my_iterator, '__next__'))  # Output: True (it's an iterator)
print(next(my_iterator))                # Output: 10 (first value)
print(next(my_iterator))                # Output: 20 (second value)

Key relationship: Every iterator is also an iterable (it has an __iter__() method that returns itself), but not every iterable is an iterator. A list is an iterable but not an iterator; calling iter() on it returns an iterator.

The Iterator Protocol in Action​

Let's see the protocol in detail with a real example:

my_list = ['a', 'b', 'c']

# Step 1: Get an iterator from the iterable
my_iterator = iter(my_list)
# This calls my_list.__iter__(), which returns a list_iterator object

print(f"Type of my_list: {type(my_list)}")
# Output: <class 'list'>

print(f"Type of my_iterator: {type(my_iterator)}")
# Output: <class 'list_iterator'>

# Step 2: Call next() on the iterator to retrieve items one by one
print(next(my_iterator))  # Output: a
print(next(my_iterator))  # Output: b
print(next(my_iterator))  # Output: c

# Step 3: When all items are exhausted, next() raises StopIteration
try:
    next(my_iterator)
except StopIteration:
    print("All items have been exhausted!")
    # Output: All items have been exhausted!

Under the hood: When you write for letter in my_list:, Python does exactly this—it gets an iterator and calls next() until StopIteration is raised:

# What the for loop does internally:
my_iterator = iter(my_list)
try:
    while True:
        letter = next(my_iterator)
        print(letter)
except StopIteration:
    pass  # Loop exits cleanly

Why This Matters: Memory Efficiency​

The iterator protocol is memory-efficient. Instead of creating a list of all values upfront (which requires allocating memory), an iterator produces values on demand. This is called lazy evaluation.

# Example: range() is an iterable, not a list
big_range = range(1_000_000)  # Creates a range object (minimal memory)

# Contrast with:
big_list = list(range(1_000_000))  # Creates a list of 1 million integers (lots of memory!)

# Iterating over big_range is just as fast as big_list, but uses far less memory
for num in big_range:
    if num > 100:
        break

According to Python documentation, range(1_000_000) uses ~28 bytes; list(range(1_000_000)) uses several megabytes. This is why iterators are preferred for large datasets.

Building a Custom Iterator​

The best way to understand the protocol is to implement it yourself. Let's create a class that acts like range(), counting up from a start value to an end value:

class Counter:
    """A simple iterator that counts from start to end."""
    
    def __init__(self, start, end):
        """Initialize the counter with start and end values."""
        self.current = start
        self.end = end
    
    def __iter__(self):
        """Make this class an iterable by returning itself as an iterator."""
        return self
    
    def __next__(self):
        """Make this class an iterator by implementing __next__."""
        if self.current >= self.end:
            # Signal that iteration is complete
            raise StopIteration
        else:
            # Return current value and prepare the next one
            value = self.current
            self.current += 1
            return value

# --- Using our custom iterator ---
my_counter = Counter(5, 8)

# Because it's an iterable, we can use it in a for loop!
for num in my_counter:
    print(num)
# Output:
# 5
# 6
# 7

Step-by-step breakdown:

  1. __init__(self, start, end): Stores the start and end values. self.current tracks the current position.

  2. __iter__(self): Returns self because our Counter object is both iterable and iterator. It has both __iter__() and __next__() methods.

  3. __next__(self): Checks if self.current >= self.end. If yes, raises StopIteration to signal the end. Otherwise, returns the current value and increments self.current for the next call.

When we call for num in my_counter:, Python:

  1. Calls iter(my_counter) → Counter.__iter__() → returns my_counter itself
  2. Calls next(my_counter) repeatedly → invokes Counter.__next__()
  3. Catches StopIteration and exits cleanly

Advanced: Separating Iterable and Iterator Classes​

In the example above, the same class was both iterable and iterator. For clarity and flexibility, you can separate them:

class CounterIterable:
    """An iterable that returns a new iterator each time."""
    
    def __init__(self, start, end):
        self.start = start
        self.end = end
    
    def __iter__(self):
        """Return a fresh iterator."""
        return CounterIterator(self.start, self.end)

class CounterIterator:
    """The actual iterator that does the counting."""
    
    def __init__(self, start, end):
        self.current = start
        self.end = end
    
    def __iter__(self):
        """Iterators are also iterable; they return themselves."""
        return self
    
    def __next__(self):
        if self.current >= self.end:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

# --- Usage ---
counter = CounterIterable(5, 8)

# You can iterate multiple times, each with a fresh iterator
for num in counter:
    print(num)  # 5, 6, 7

for num in counter:
    print(num)  # 5, 6, 7 again

Advantage: If you call iter() multiple times on CounterIterable, you get fresh iterators, not the same exhausted iterator.

Frequently Asked Questions​

What happens if I call next() on an exhausted iterator?​

It raises StopIteration every time until you get a fresh iterator:

my_iterator = iter([1, 2, 3])
next(my_iterator)  # 1
next(my_iterator)  # 2
next(my_iterator)  # 3
next(my_iterator)  # StopIteration
next(my_iterator)  # StopIteration (again)

To iterate again, you need a fresh iterator:

my_iterator = iter([1, 2, 3])  # New iterator
next(my_iterator)  # 1

Can I use list() to consume an iterator?​

Yes. list() calls next() repeatedly until StopIteration:

my_iterator = iter([10, 20, 30])
my_list = list(my_iterator)
print(my_list)  # [10, 20, 30]

# Iterator is now exhausted
next(my_iterator)  # StopIteration

What is the difference between an iterator and a generator?​

A generator is a special function that returns an iterator. It uses the yield keyword instead of return. Generators are a more concise way to create iterators than defining a class. We'll cover generators in the next article.

Can iterators be reset?​

No, built-in iterators cannot be reset once exhausted. You must get a fresh iterator from the original iterable:

my_list = [1, 2, 3]
iterator1 = iter(my_list)
list(iterator1)  # Consumes it
next(iterator1)  # StopIteration

# Get a fresh iterator
iterator2 = iter(my_list)
next(iterator2)  # 1 (fresh start)

What is the performance overhead of iterators vs. lists?​

For small datasets, negligible. For large datasets, iterators save massive amounts of memory. A generator that yields 1 billion values uses constant memory; a list of 1 billion values would require gigabytes. Always prefer iterators for large or unbounded sequences.

Practice Challenge​

Create a custom iterator class called FibonacciIterator that yields Fibonacci numbers up to a given limit:

  • First two Fibonacci numbers: 0, 1
  • Each subsequent number is the sum of the previous two: 1, 1, 2, 3, 5, 8, 13, ...

Your iterator should yield numbers until they exceed the limit.

Solution:

class FibonacciIterator:
    """Yields Fibonacci numbers up to a given limit."""
    
    def __init__(self, limit):
        self.limit = limit
        self.prev, self.curr = 0, 1
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.curr > self.limit:
            raise StopIteration
        value = self.curr
        self.prev, self.curr = self.curr, self.prev + self.curr
        return value

# Test
for num in FibonacciIterator(100):
    print(num, end=' ')
# Output: 1 1 2 3 5 8 13 21 34 55 89

Conclusion​

The iterator protocol is a cornerstone of Python's design. It provides a unified, memory-efficient way to iterate over any sequence or data source. By understanding the distinction between iterables and iterators, and how __iter__() and __next__() work, you unlock the ability to:

  • Create memory-efficient custom sequences
  • Understand how for loops work under the hood
  • Build the foundation for generators and comprehensions
  • Write Pythonic code that other developers will instantly understand

Key takeaways:

  • An iterable has __iter__() and returns an iterator.
  • An iterator has __next__() and tracks state.
  • The for loop calls iter() then next() until StopIteration.
  • Iterators enable lazy evaluation—values produced on demand, not upfront.
  • Custom iterators let you create any sequence imaginable.

Next Steps​

While creating a class-based iterator is powerful, it can be verbose for simple cases. Python provides a much more concise and elegant way to create iterators called generators, which use the yield keyword. In the next article, explore "Generators: Creating iterators with yield" to see a shorthand that accomplishes the same goals with less boilerplate.

Further Reading​