Python Generators — The Empty Log Report Bug

Every Python developer hits a wall: they write a reasonable script that loads a dataset, processes it, and crashes — not because the logic is wrong, but because they tried to hold a million rows in memory all at once. It's one of the most common avoidable performance problems, and generators exist to solve it. They're not niche; they power Python's own range(), map(), and zip().

The core problem is the cost of 'eagerness'. A regular list computes and stores every value immediately. Fine for 100 items. A disaster for 10 million log entries, infinite sequences, or streaming API responses where you don't even know the final count. Generators flip the model: they're lazy, producing each value only when the caller asks for the next one. Memory stays flat no matter how large the dataset.

By the end you'll understand why yield exists and how it differs from return, you'll write generator functions and generator expressions with confidence, and you'll know the real-world patterns — log file processing, data pipelines, infinite sequences — where generators genuinely shine. You'll also avoid the two traps that catch almost every developer the first time.

What yield Actually Does — and Why It's Not Just a Fancy return

The single most important thing to understand about generators is what happens to the function's execution state when it hits yield. With a normal return, the function runs, hands back a value, and is completely torn down — local variables gone, position in code gone, everything erased. When a function hits yield, Python does something different: it pauses the function, hands the yielded value to the caller, and freezes the entire execution frame in place — local variables, loop counters, everything. The next time the caller asks for a value by calling next(), Python thaws that frozen frame and continues from the exact line after yield.

This is why a generator function doesn't execute at all when you call it. Calling a generator function just returns a generator object. The body doesn't run until you start consuming that object with next() or a for loop. That single distinction trips up almost every developer the first time.

import sys

# io.thecodeforge — Basic Generator Implementation
def count_up_to(maximum):
    current = 1
    while current <= maximum:
        # State is frozen right here
        yield current
        # Resumes here on the next call
        current += 1

# 1. Calling the function returns the object, does NOT execute the body
counter = count_up_to(5)
print(f"Object type: {type(counter)}")

# 2. Manual consumption
print(f"First value: {next(counter)}")

# 3. Iteration (handles StopIteration automatically)
for number in counter:
    print(f"Iterated: {number}")

# 4. Memory comparison
eager_list = [i for i in range(10000)]
lazy_gen = (i for i in range(10000))

print(f"List Size: {sys.getsizeof(eager_list)} bytes")
print(f"Gen Size:  {sys.getsizeof(lazy_gen)} bytes")

Real-World Pattern — Processing Large Log Files

Log files are the textbook generator use case because they're naturally sequential and can grow into the gigabytes. Loading a 10 GB file into a list will crash most systems, but a generator pipeline handles it with a constant memory footprint. The pattern involves 'pipelining' where each step is a generator that pulls from the previous one, ensuring only one line of data exists in RAM at any given time.

By decoupling the reading, filtering, and parsing logic into separate generator functions, you create a modular, production-grade ETL (Extract, Transform, Load) system that is as readable as it is efficient.

import os

def get_log_lines(filename):
    """Generator to stream lines from a file."""
    with open(filename, 'r') as f:
        for line in f:
            yield line.strip()

def filter_errors(lines):
    """Generator to filter for ERROR status."""
    for line in lines:
        if "ERROR" in line:
            yield line

def parse_details(error_lines):
    """Generator to extract specific error messages."""
    for line in error_lines:
        yield line.split(" : ")[-1]

# Building the Pipeline (No execution yet)
# raw_log = get_log_lines("production_log.txt")
# errors = filter_errors(raw_log)
# final_report = parse_details(errors)

# for msg in final_report:
#     print(f"Critial Alert: {msg}")

Memory Usage Comparison: List vs Generator

One of the most compelling reasons to use generators is the dramatic difference in memory consumption. A list stores every element in contiguous memory. A generator stores nothing — it computes each element on demand and discards it after yielding. This comparison table crystallizes the practical trade-offs for common Python workloads.

For a dataset of 10 million integers, a list would consume roughly 80 MB (8 bytes per int * 10M + list overhead). A generator requires just 112 bytes — the size of the generator object itself. The speed difference is negligible for iteration (generators add about 40ns per yield), but the memory savings are enormous.

The table below summarizes the key differences for production decision-making:

import sys
import tracemalloc

N = 10_000_000

# Tracemalloc to measure peak memory
tracemalloc.start()

# List version
eager = [i for i in range(N)]
current, peak = tracemalloc.get_traced_memory()
print(f"List: Current = {current / 1024**2:.2f} MB, Peak = {peak / 1024**2:.2f} MB")
tracemalloc.stop()

tracemalloc.start()
# Generator version
lazy = (i for i in range(N))
# Simulate consumption without storing
for _ in lazy:
    pass
current, peak = tracemalloc.get_traced_memory()
print(f"Generator: Current = {current / 1024**2:.2f} MB, Peak = {peak / 1024**2:.2f} MB")
tracemalloc.stop()

Advanced Mechanics: Infinite Streams and .send()

Because generators are lazy, they are the only way to represent infinite sequences. A while True loop inside a generator isn't a bug—it's a feature. Since the function pauses at every yield, it will never hang your CPU; it simply waits for the caller to request the next value. Furthermore, the .send() method allows you to push data into the generator, effectively turning it into a coroutine for two-way communication.

def infinite_fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

def tally_tracker():
    """Receives values and yields the current sum."""
    total = 0
    while True:
        val = yield total
        if val is not None:
            total += val

# Infinite usage
fib = infinite_fibonacci()
print([next(fib) for _ in range(5)])

# Two-way usage
stats = tally_tracker()
next(stats) # Prime the generator
print(f"Total after 10: {stats.send(10)}")
print(f"Total after 25: {stats.send(25)}")

yield from — Generator Delegation Made Simple

When you have nested generators, you could write a for loop to yield all items from a sub-generator. But yield from does it cleaner and faster. It delegates to another generator (or any iterable) and yields each item as if it came from the outer generator. It also propagates StopIteration and handles .send() and .throw() correctly — something a for loop doesn't do.

Use yield from when you need to flatten nested data, compose generators, or build recursive generators. It's the unsung hero of lazy pipeline designs.

def sub_gen():
    yield "A"
    yield "B"

def main_gen():
    yield "Start"
    yield from sub_gen()  # delegates to sub_gen
    yield "End"

for item in main_gen():
    print(item)

# Recursive example: flatten nested lists lazily
def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)  # recurse through sublists
        else:
            yield item

nested_list = [1, [2, [3, 4], 5], 6]
flat = flatten(nested_list)
print(list(flat))  # [1, 2, 3, 4, 5, 6]

yield from for Recursive and Nested Generators

While the basic yield from works for simple delegation, its real power emerges when you need to recursively traverse deeply nested structures. Consider a file system tree, a JSON object with arbitrary nesting, or a game tree. A generator that recursively yields from sub-generators lets you produce a flat stream of elements without building intermediate lists.

The recursive pattern works because each call to yield from flatten(...) creates a new generator that yields items one by one. Python's call stack pushes frames for each level of nesting, but only one value exists at a time. This is a textbook example of lazy recursion: you can flatten a tree of any depth without running out of memory (though you can hit recursion depth limits for very deep trees).

For production use, combine this with itertools.islice or itertools.takewhile to limit output when you only need a subset of the nested data.

import os

# Recursively walk a directory tree, yielding file paths lazily
def walk_files(root):
    for entry in os.scandir(root):
        if entry.is_dir():
            yield from walk_files(entry.path)
        else:
            yield entry.path

# Usage: only one file path in memory at a time
# for file_path in walk_files("/var/log"):
#     process(file_path)

# Another pattern: nested JSON traversal
def extract_strings(data):
    if isinstance(data, dict):
        for _, v in data.items():
            yield from extract_strings(v)
    elif isinstance(data, list):
        for item in data:
            yield from extract_strings(item)
    elif isinstance(data, str):
        yield data

nested = {
    "a": "hello",
    "b": {"c": ["world", "foo"]},
    "d": [1, {"e": "bar"}]
}
print(list(extract_strings(nested)))
# ['hello', 'world', 'foo', 'bar']

Generators vs Lists vs Iterators — Knowing When to Use Each

The honest answer to 'when should I use a generator?' is: whenever you don't need all the values at once, or whenever you might not need all of them at all. If you need to sort, reverse, index by position, or pass the same sequence to multiple consumers, use a list — you need all values materialised. If you're transforming or filtering a sequence and consuming it exactly once from start to finish, a generator is almost always the better choice.

One critical difference that surprises people: generators are single-use. Once exhausted, they're done — calling iter() on them again doesn't restart them. A list can be iterated as many times as you like. This is the most common source of subtle bugs with generators in production code.

Custom iterator classes (with __iter__ and __next__) give you the same lazy behaviour as generators but with more control — you can maintain state, support multiple independent iterations, or define a length. Generators are the shortcut for the 80% case where you just need simple, one-shot lazy iteration.

def get_gen():
    yield 1
    yield 2

my_gen = get_gen()

# Pass 1
print(list(my_gen)) # [1, 2]

# Pass 2
print(list(my_gen)) # [] - The generator is empty!

# Pass 3: Re-calling the function creates a FRESH generator
fresh_gen = get_gen()
print(list(fresh_gen)) # [1, 2]

Advanced Generator Methods: .send() and .throw()

Beyond simple iteration, generators support two advanced methods that turn them into two-way communication channels: .send() and .throw(). These are often overlooked but essential for building coroutine-like patterns, cooperative multitasking, and generator-based pipelines with error handling.

.send(val) resumes the generator and passes a value into it, which becomes the result of the yield expression inside the generator. This lets you inject data from outside. .throw(type, value, traceback) raises an exception at the point where the generator was paused. The generator can catch it (via try/except around the yield) and yield another value, or let it propagate to terminate the generator.

A common use case for .throw() is to signal a generator to clean up or stop early, akin to a cancel signal. For pipelines, you can throw an exception into the middle of a chain to abort processing without manually draining the generator.

# .send() example
from typing import Generator

def accumulator() -> Generator[float, float, None]:
    total = 0.0
    while True:
        increment = yield total
        if increment is not None:
            total += increment

acc = accumulator()
next(acc)  # prime
print(acc.send(10.5))  # 10.5
print(acc.send(3.2))   # 13.7


# .throw() example – stopping a generator gracefully
def produce_items():
    try:
        for i in range(1000):
            yield i
    except GeneratorExit:
        print("Cleanup: closing generator")
    except Exception as e:
        print(f"Caught exception: {e}")

producer = produce_items()
print(next(producer))  # 0
print(next(producer))  # 1
producer.throw(RuntimeError, "cancel")
# output: Caught exception: cancel
# generator ends

# Without catching, .throw() propagates
def simple_gen():
    yield 1
    yield 2

g = simple_gen()
next(g)
try:
    g.throw(ValueError, "test")
except ValueError:
    print("ValueError propagated")

Generator Expressions: The One-Liner That Saves Your Stack

You've used list comprehensions. They're clean, readable, and will crash your box on a 10GB dataset. Generator expressions do the same thing without materializing the entire list. The syntax is almost identical — swap square brackets for parentheses.

But here's the catch: generator expressions are single-pass. You can't index them, you can't slice them, and once you've consumed them they're gone. This isn't a bug — it's the whole point. You trade random access for memory efficiency that scales to any dataset size.

The real power comes from chaining them. A pipeline of generator expressions processes data in a single pass without intermediate storage. Three comprehension-like transformations? That's three generator expressions linked together, streaming elements one at a time. No intermediate lists, no memory spikes, no surprise OOM kills in production.

// io.thecodeforge — python tutorial

def read_sensor_log(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

raw = read_sensor_log("/var/log/sensors/temperature.csv")

# Three transformations, zero intermediate lists
readings = (line.split(",") for line in raw)
validated = (r for r in readings if len(r) == 3)
temperatures = (float(r[1]) for r in validated if r[2] == "OK")

# This single loop streams through all three stages
for temp in temperatures:
    if temp > 85.0:
        print(f"ALERT: temperature {temp} exceeds threshold")

Profiling Generator Performance — When Lazy Isn't Faster

Developers assume generators are always faster because they're memory-efficient. That's wrong. Generators have overhead: function call state tracking, yield/resume cycles, and the context switch between iterations. For small datasets, a list comprehension beats a generator expression every time. The question is where the crossover point lives.

List comprehension: allocate a list, compute all values, return. If you only need 5 items from a 10,000-element collection, that's 9,995 wasted computations. Generator: compute one value, yield, pause. If you break early, you skip the rest. Zero waste.

But if you iterate every single element and the computation per element is trivial — say a simple integer operation — the list's lower per-element overhead wins. The generator's yield machinery adds microseconds per iteration. On a million elements, microseconds add up to seconds.

The rule: benchmark with your actual data shape. Profile before you optimize. And never replace a list comprehension with a generator expression just because someone on Reddit said it's "better." It's only better when you're memory-bound or you won't consume the entire sequence.

// io.thecodeforge — python tutorial

import time
import sys

def generate_ids(n):
    for i in range(n):
        yield i * 2

def list_ids(n):
    return [i * 2 for i in range(n)]

n = 10_000_000

# Generator — lazy, memory-light
start = time.perf_counter()
gen = generate_ids(n)
for val in gen:
    _ = val  # simulate processing each item
print(f"Generator: {time.perf_counter() - start:.2f}s")

# List — eager, memory-heavy
start = time.perf_counter()
lst = list_ids(n)
for val in lst:
    _ = val
print(f"List:       {time.perf_counter() - start:.2f}s")

send() Is How You Talk Back to a Generator

Most devs treat generators as one-way data pipes. You call next(), you get a value. That's fine for iterating over log files. But generators can receive data mid-execution using .send(). This turns them into coroutines — lightweight cooperative threads.

The trick: .send() resumes the generator AND injects a value into the yield expression. The first call MUST be next() or send(None) because no yield has been hit yet. After that, each send(val) sets yield's return value. This is how you implement state machines, streaming pipelines, or cooperative task schedulers without threading overhead.

Why bother? Because you avoid global state, external queues, and callback hell. The generator keeps its own context on the stack. Send data in, get data out. Clean, testable, production-hardened.

// io.thecodeforge — python tutorial

def running_average():
    total = 0.0
    count = 0
    average = None
    while True:
        new_value = yield average
        if new_value is None:
            continue
        total += new_value
        count += 1
        average = total / count

avg_gen = running_average()
next(avg_gen)                # prime it
print(avg_gen.send(10))      # 10.0
print(avg_gen.send(20))      # 15.0
print(avg_gen.send(30))      # 20.0

close() Is How You Fire a Generator Cleanly

Generators hold resources: open file handles, database cursors, socket connections. If you stop iterating early — break out of a for loop, raise an exception — the generator's stack frame freezes. That file handle stays open until garbage collection kicks in. That's a leak waiting to happen in production.

.close() raises GeneratorExit inside the generator at its current yield point. If your generator has a try/finally block, that finally runs. No other exception is raised to the caller. It's a clean, deterministic shutdown.

Pair this with contextlib.closing() or wrap your generator in a context manager. Never rely on gc to clean up your I/O. Explicit shutdown beats implicit leaks every time. Treat .close() like closing a file handle — you don't walk away leaving files open, don't walk away leaving generators open.

// io.thecodeforge — python tutorial

def read_lines(file_path):
    try:
        f = open(file_path, 'r')
        for line in f:
            yield line.strip()
    finally:
        f.close()
        print('File closed')

gen = read_lines('/etc/hostname')
print(next(gen))
gen.close()
print('Generator closed, file released')