Python generators & iterators

Iterables vs iterators: the protocol

An iterable is any object that can hand out its items one at a time: it defines __iter__ (or the older sequence __getitem__), and iter(obj) returns an iterator. An iterator is the object that walks the stream. It defines __next__, which returns the next value or raises StopIteration when the data runs out, and its __iter__ returns itself.

That self-returning __iter__ is why iterators are single-use, while containers are reusable.

nums = [1, 2, 3]
it = iter(nums)      # a fresh list_iterator over nums
next(it)             # 1
next(it)             # 2
iter(it) is it       # True: an iterator is its own iterator
iter(nums) is nums   # False: a list hands out a new iterator each time

A for loop calls iter(obj) once, then next() until StopIteration, which it swallows quietly.

Writing a generator with yield

Any def that contains yield is a generator function. Calling it runs none of the body: it returns a generator object. Each next() runs the body until the next yield, hands back that value, and suspends there, keeping local variables intact. The following next() resumes on the line right after the yield.

def countdown(n):
    while n > 0:
        yield n          # hand back n, then pause here
        n -= 1

list(countdown(3))       # [3, 2, 1]

No class, no __iter__/__next__, no manual state variable: the function's own locals and instruction pointer are the state.

Wrong "When a generator runs out, next() returns None." It raises StopIteration. Right to get a fallback instead of an exception, pass one: next(g, "done") returns "done" once the generator is exhausted.

Generator expressions vs list comprehensions

Parentheses build a generator expression, which is a lazy iterator. Brackets build a list comprehension, which eagerly constructs the whole list in memory.

squares = (x * x for x in range(5))   # a generator; nothing computed yet
sum(x * x for x in range(5))          # 30; parens optional as the sole argument
[x * x for x in range(5)]             # [0, 1, 4, 9, 16]; built up front

A genexpr holds only a paused frame, so it stays small no matter how many items it will yield. Being an iterator, it is single-use too: iterate it twice and the second pass is empty.

itertools for lazy pipelines

itertools gives you building blocks that stream instead of materializing. They return iterators, so you can chain them over huge or infinite sources and pull items through one at a time.

from itertools import count, islice, chain
list(islice(count(10), 5))     # [10, 11, 12, 13, 14]
list(chain([1, 2], [3, 4]))    # [1, 2, 3, 4]

Reach for a list when you need len, indexing, or more than one pass; reach for a generator when the data is large, infinite, or read once.

Generators as coroutines: send, throw, close

send(value) resumes a paused generator and makes the yield expression it was sitting on evaluate to value, then returns the next yielded value. There is a priming rule: the first activation of a fresh generator must be next(g) or g.send(None), because there is no paused yield yet to receive a value. g.send(42) on a just-started generator raises TypeError: can't send non-None value to a just-started generator.

def running_sum():
    total = 0
    while True:
        x = yield total   # None when resumed by next(), the sent value under send()
        total += x

g = running_sum()
next(g)        # 0  (prime)
g.send(10)     # 10
g.send(5)      # 15

throw(exc) raises an exception at the pause point, which a try/except inside the generator can catch. The three-argument throw(type, value, traceback) form is deprecated since Python 3.12; pass an exception instance instead. close() raises GeneratorExit at the pause point, so a finally or with block runs its cleanup. Since Python 3.13, if the body catches GeneratorExit and returns a value, close() returns that value; earlier versions always returned None.

yield from: delegating to a sub-iterable

RESULT = yield from EXPR drives any iterable and passes its values straight to the caller. It also forwards send, throw, and close into the subgenerator, which a hand-written for x in it: yield x loop does not. When the subgenerator finishes, its return value becomes the value of the yield from expression.

def sub():
    yield "a"
    yield "b"
    return "sub-done"      # surfaces as StopIteration.value

def deleg():
    result = yield from sub()   # result == "sub-done"
    yield result

list(deleg())      # ['a', 'b', 'sub-done']

A return value in any generator is equivalent to raise StopIteration(value) (PEP 380, Python 3.3), so the value rides on StopIteration.value and is otherwise easy to drop on the floor.

groupby needs sorting and shares its source

groupby yields (key, group) pairs over consecutive runs of equal keys. It does not aggregate globally like SQL GROUP BY, so unsorted input fragments into repeated keys. Sort on the same key first.

from itertools import groupby
key = lambda w: w[0]
words = ["apple", "apricot", "banana", "avocado"]
[k for k, _ in groupby(words, key)]           # ['a', 'b', 'a']
[k for k, _ in groupby(sorted(words), key)]   # ['a', 'b']

The group iterator shares the underlying source, so it goes stale the moment you advance to the next group.

Wrong pairs = [(k, g) for k, g in groupby(sorted(words), key)], then reading a g later. Every group comes back empty, because advancing the groupby object invalidated the previous one. Right materialize inside the loop with (k, list(g)) before moving on.

When StopIteration escapes: PEP 479

A bare next(it) inside a generator body looks harmless until the inner iterator runs dry. In current Python a StopIteration that escapes a generator is converted to RuntimeError (the PEP 479 rule), so this crashes rather than silently stopping the outer generator early.

def take_pairs(it):
    it = iter(it)
    while True:
        yield next(it), next(it)   # a bare next() can raise StopIteration

list(take_pairs([1, 2, 3]))   # RuntimeError: generator raised StopIteration

Catch StopIteration locally or return from the generator instead of letting it propagate. The same rule is why a live generator holds a real stack frame: leaving it un-exhausted keeps its locals and any open resources alive until close() or garbage collection runs, which is why cleanup belongs in a try/finally inside the generator.