Python Iterator Exhaustion — Silent Data Drop in ETL

Every Python developer uses for-loops from day one, but almost nobody stops to ask: how does Python actually know what to do next on each loop cycle? The answer lives inside a two-method protocol — __iter__ and __next__ — and once you understand it, you'll see it everywhere: in file reading, database cursors, API pagination, and streaming data pipelines. This isn't just academic knowledge; it's the engine under the hood of the language itself.

The problem this solves is memory and control. If Python loaded every item from a collection into memory before you could loop over it, working with a 10-million-row CSV file or an infinite sequence of sensor readings would be impossible. Iterators let you process one item at a time, on demand, without ever needing to know how many items exist in total. This lazy evaluation is what makes Python practical for real data-engineering work.

By the end of this article you'll be able to explain exactly what happens when Python executes a for-loop, write your own custom iterator class from scratch, spot the difference between an iterable and an iterator in a code review, and avoid the subtle bugs that trip up even experienced developers when they assume an iterator can be reused.

Why Iterator Exhaustion Is a Silent Data Bug

An iterator in Python is an object that implements __iter__() and __next__(), producing values one at a time and raising StopIteration when exhausted. An iterable is any object that can return an iterator via iter(). The core mechanic: iterators are stateful — they track position and can only be traversed once. This single-pass design is memory-efficient (O(1) space) but introduces a hidden failure mode when code assumes reusability.

In practice, passing an iterator to multiple consumers (e.g., two list() calls, or a for loop followed by sum()) silently yields empty results after the first consumption. The iterator doesn't reset; it stays exhausted. This contrasts with iterables like lists, which produce fresh iterators each time. The distinction matters because many built-in functions (map, filter, zip) and generators return iterators, not iterables.

Use iterators when processing large or infinite streams where memory is constrained. But never assume an iterator can be reused. In ETL pipelines, this mistake drops entire datasets without errors — the second consumer simply sees zero rows. Always convert to a concrete collection (list, tuple) if you need multiple passes, or restructure to a single-pass pattern.

The Two-Protocol System: Iterable vs Iterator

Python draws a firm line between two roles. An iterable is any object that knows how to produce an iterator — it has an __iter__ method that returns one. A list, a string, a tuple, a dict — all iterables. An iterator is the stateful worker that actually does the traversal. It has both __iter__ (which just returns itself) and __next__ (which delivers the next item or raises StopIteration when it's done).

This separation exists for a good reason: you want to be able to loop over the same list a hundred times without it 'running out'. The list (iterable) stays neutral. Each time you start a new loop, Python silently calls iter(your_list) to create a fresh iterator — a brand-new dealer for your deck of cards.

You can manually step through this process with Python's built-in iter() and next() functions, which is exactly what a for-loop does internally on every single iteration. Understanding this unlocks everything else in this article.

# A plain list is an ITERABLE — it can produce iterators on demand
playlist = ["Bohemian Rhapsody", "Hotel California", "Stairway to Heaven"]

# iter() calls playlist.__iter__() and returns a fresh ITERATOR
playlist_iterator = iter(playlist)

# next() calls playlist_iterator.__next__() each time
print(next(playlist_iterator))  # Fetches item 1 — iterator remembers position
print(next(playlist_iterator))  # Fetches item 2 — picks up exactly where it left off
print(next(playlist_iterator))  # Fetches item 3 — last item

# The iterator is now exhausted — calling next() again raises StopIteration
try:
    print(next(playlist_iterator))
except StopIteration:
    print("Iterator exhausted — no more songs!")

# The original list is UNCHANGED — start a fresh iterator anytime
fresh_iterator = iter(playlist)
print(next(fresh_iterator))  # Back to the beginning — "Bohemian Rhapsody"

# A for-loop does ALL of this invisibly on every iteration
for song in playlist:          # Python calls iter(playlist) once here
    print(f"Now playing: {song}")  # Then calls next() on each cycle

Building a Custom Iterator — A Real-World File Chunker

Here's where things get genuinely useful. Let's say you're processing a large log file and you want to read it in fixed-size chunks rather than line by line or all at once. You can't do this elegantly with a plain list. This is the exact scenario custom iterators were made for.

To make an object an iterator, you implement two methods: __iter__ returns self (because the iterator is its own iterable), and __next__ returns the next value or raises StopIteration. That's the entire contract.

The power here is state. Your iterator class can carry any state it needs between calls to __next__ — a file handle, a counter, a buffer, a database cursor position. This is what separates a custom iterator from a simple function: it pauses between calls and picks up exactly where it left off, making it perfect for streaming, pagination, and lazy computation.

class FileChunkIterator:
    """
    Reads a text file in fixed-size chunks.
    Useful for processing large files without loading them entirely into memory.
    """

    def __init__(self, filepath, chunk_size=5):
        self.filepath = filepath
        self.chunk_size = chunk_size  # How many lines per chunk
        self._file_handle = None      # Will hold the open file — initialised in __iter__
        self._line_buffer = []        # Accumulates lines until chunk is full

    def __iter__(self):
        # Open the file fresh each time iteration starts
        # This allows the iterator to be restarted cleanly
        self._file_handle = open(self.filepath, "r")
        self._line_buffer = []
        return self  # An iterator must return itself here

    def __next__(self):
        # Try to fill a chunk from the file
        while len(self._line_buffer) < self.chunk_size:
            raw_line = self._file_handle.readline()
            if not raw_line:  # readline() returns '' at end of file
                break
            self._line_buffer.append(raw_line.strip())

        if not self._line_buffer:  # Buffer empty — file is fully consumed
            self._file_handle.close()  # Clean up the file handle
            raise StopIteration      # Signal to the for-loop: we're done

        # Pull exactly chunk_size lines (or fewer if we're near the end)
        chunk = self._line_buffer[:self.chunk_size]
        self._line_buffer = self._line_buffer[self.chunk_size:]
        return chunk


# --- Demo: create a temporary log file and iterate over it in chunks ---
import tempfile
import os

# Write 12 fake log lines to a temp file
log_lines = [f"[INFO] Event #{i} processed successfully" for i in range(1, 13)]

with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as temp_log:
    temp_log.write("\n".join(log_lines))
    temp_filepath = temp_log.name

# Iterate over the log in chunks of 5 lines
chunk_reader = FileChunkIterator(temp_filepath, chunk_size=5)

for chunk_number, log_chunk in enumerate(chunk_reader, start=1):
    print(f"--- Chunk {chunk_number} ({len(log_chunk)} lines) ---")
    for log_entry in log_chunk:
        print(f"  {log_entry}")

os.unlink(temp_filepath)  # Clean up the temp file

Custom Iterator Class Implementation — Step-by-Step Template

While the file chunker above is a practical example, you'll often need a generic blueprint for building your own custom iterators. The pattern is always the same, whether you're wrapping a database cursor, a paginated API, or a live data stream.

Step 1: Define the class and __init__ – Accept your data source and any configuration. Store everything you need to start fresh. Do not open external resources here yet — delay that to __iter__ to allow restartability.

Step 2: Implement __iter__ – This method returns the iterator object. For one-shot iterators, return self. If you want the iterator to be restartable (i.e., you can call iter() again and get a fresh start), reset all state here, re-open resources, and return self. Restartability is optional but powerful.

Step 3: Implement __next__ – This is the core. Check if there's a next item available. If yes, return it after advancing internal state. If no, raise StopIteration. Every code path must end with either a return or a StopIteration — no fall-through.

Step 4: Handle cleanup – Close files, connections, or release locks when StopIteration is raised. Alternatively, implement __del__ as a safety net, but don't rely on it solely because garbage collection timing is unpredictable.

Step 5: Test with edge cases – Empty data, single item, partial iteration (breaking out early), and multiple iterations (if restartable).

class GenericIterator:
    """
    Template for a restartable custom iterator.
    Adapt __init__ to your data source and __next__ to your traversal logic.
    """
    def __init__(self, data_source):
        # Source could be a list, file path, API client, etc.
        self._source = data_source
        # Internal state (will be reset in __iter__ if restartable)
        self._position = 0
        self._resource = None

    def __iter__(self):
        # Reset state so we can iterate again
        self._position = 0
        # Re-open external resource (e.g., file, network socket)
        # self._resource = open(self._source, 'r')
        return self

    def __next__(self):
        # 1. Check stop condition
        if self._position >= len(self._source):  # example for a list
            # Cleanup before raising
            # if self._resource:
            #     self._resource.close()
            raise StopIteration
        # 2. Fetch next item
        item = self._source[self._position]
        # 3. Advance state
        self._position += 1
        return item

# Usage
my_iter = GenericIterator([10, 20, 30])
for val in my_iter:
    print(val)
print("Second pass:")
for val in my_iter:  # Works because __iter__ resets position
    print(val)

Generator Functions — The Shortcut Python Gives You

Writing a full iterator class is powerful, but verbose. Python gives you a shortcut: generator functions. Any function that contains a yield statement automatically becomes a factory for iterator objects called generators. Python handles all the __iter__ and __next__ plumbing for you.

Under the hood, calling a generator function doesn't execute the body at all — it returns a generator object. Each call to next() on that object resumes execution from the last yield, suspending again at the next one. This is exactly the same pause-and-resume behaviour as our custom iterator, but expressed in a fraction of the code.

The real-world sweet spot for generators is producing sequences that are either very large or computationally expensive — think paginated API responses, mathematical series, or streaming transformations. If your data source is 'pull-based' (you ask for the next item when you're ready), a generator is almost always the right tool.

import time

# Simulates a paginated REST API that returns users in pages
def mock_api_fetch(page_number, page_size=3):
    """
    Pretend this is calling requests.get('https://api.example.com/users?page=N').
    Returns a list of users for that page, or empty list when pages run out.
    """
    all_users = [
        {"id": 1, "name": "Alice Nakamura"},
        {"id": 2, "name": "Ben Okafor"},
        {"id": 3, "name": "Carmen Reyes"},
        {"id": 4, "name": "David Chen"},
        {"id": 5, "name": "Elena Petrov"},
        {"id": 6, "name": "Farhan Malik"},
        {"id": 7, "name": "Grace Osei"},
    ]
    start_index = (page_number - 1) * page_size
    return all_users[start_index : start_index + page_size]


def paginated_user_stream(page_size=3):
    """
    Generator that fetches users page-by-page.
    Callers never need to know pagination exists — they just iterate.
    """
    current_page = 1
    while True:
        print(f"  [Generator] Fetching page {current_page} from API...")
        page_data = mock_api_fetch(current_page, page_size)

        if not page_data:  # Empty page means no more data
            print("  [Generator] All pages consumed — raising StopIteration")
            return  # 'return' inside a generator raises StopIteration automatically

        for user in page_data:
            yield user  # Suspend here, hand one user to the caller

        current_page += 1  # Only advance the page after all users on it are yielded


# The caller just iterates — zero pagination logic leaks out
print("Streaming all users from API:\n")
for user in paginated_user_stream(page_size=3):
    print(f"  Processing user: {user['name']} (ID: {user['id']})")

print("\nDone — all users processed.")

itertools Quick Reference — Lazy Iterator Building Blocks

Python's itertools module is a collection of fast, memory-efficient iterator building blocks. Every function in itertools returns a lazy iterator — nothing is evaluated until you loop over it. This makes them ideal for chaining transformations without blowing up memory.

Here's a quick reference table of the most commonly used itertools functions. Use this as a cheat sheet during development:

When to use itertools? Any time you're writing a custom loop that involves skipping, grouping, or combining sequences. These functions are implemented in C and are significantly faster than equivalent pure-Python code.

import itertools

# Infinite cycling – useful for round-robin routing
colors = ['red', 'blue', 'green']
color_cycle = itertools.cycle(colors)
print("Cycle:", [next(color_cycle) for _ in range(5)])

# Accumulate – running total of sales
sales = [100, 200, 150, 300]
running = list(itertools.accumulate(sales))
print("Accumulate:", running)

# Chain – merge two log files
log1 = ['ERROR: conn failed', 'WARN: timeout']
log2 = ['INFO: retry', 'ERROR: disk full']
merged = list(itertools.chain(log1, log2))
print("Chain:", merged)

# Product – all combinations of two features
features = ['a', 'b']
flags = [1, 2]
combos = list(itertools.product(features, flags))
print("Product:", combos)

# Takewhile – read until a sentinel line with log parsing
lines = ['start', 'ok', 'END', 'more']
for line in itertools.takewhile(lambda x: x != 'END', lines):
    print("Takewhile:", line)

Lazy Evaluation — How Iterators Enable Streaming and Large Data Processing

The real superpower of iterators isn't just the protocol — it's that they evaluate values only when asked. This is lazy evaluation. Instead of building a whole list in memory, an iterator produces one element at a time. That means you can process data streams that would never fit in RAM: reading a 100 GB log file, iterating over an infinite mathematical sequence, or consuming a real-time sensor feed.

Python's standard library is full of lazy iterators: map(), filter(), zip(), enumerate(), reversed() (on sequences) — all return iterators. Even range() returns an iterable that produces numbers on demand, not a list of all numbers. This design is intentional: Python defaults to lazy unless you force it with list(), tuple(), or a comprehension with brackets.

Understanding lazy evaluation helps you design systems that are memory-efficient by default. If you find yourself calling list() on a generator just to pass it to a function, stop and ask: does that function truly need random access, or can it work with a stream? In many cases, the function itself can be refactored to iterate lazily.

# Compare memory usage between eager (list) and lazy (generator) for a range

# Eager: creates a list of 10 million integers — ~400 MB in memory!
eager_squares = [x * x for x in range(10_000_000)]  # DON'T run this unless you have spare memory

# Lazy: generator expression produces one square at a time — ~0 bytes for the data
lazy_squares = (x * x for x in range(10_000_000))

# Use a small slice to demonstrate laziness
small_lazy = (x * x for x in range(10))
for value in small_lazy:
    print(value, end=' ')  # Prints 0 1 4 9 16 25 36 49 64 81

print()

# map() and filter() are also lazy — they don't compute until iterated
nums = [1, 2, 3, 4, 5]
lazy_doubled = map(lambda n: n * 2, nums)
print(lazy_doubled)  # <map object at 0x...> — not a list!
print(list(lazy_doubled))  # [2, 4, 6, 8, 10]

# Practical example: reading lines from a huge file without loading all
# with open('giant_log.txt') as f:
#     for line in f:    # f is an iterator over lines
#         process(line)  # Only one line in memory at a time

Memory Efficiency Comparison: List vs Generator (Eager vs Lazy)

The single most important practical difference between lists and generators is memory consumption. For large datasets, a list stores all elements in memory simultaneously, while a generator produces each element on demand and discards it after use. This difference can mean the difference between a pipeline that runs on a laptop and one that crashes with MemoryError.

Below is a comparison table for a sequence of n integers (assuming Python 3.12 on a 64-bit system). Actual numbers vary by Python version and system, but the ratios hold.

As you can see, the list's memory grows linearly with n, whereas the generator object's size is constant because it doesn't store the data — only a reference to the generating function and current state.

The same applies to lazy iterator counterparts of list operations: map vs list comprehension, filter vs list comprehension, zip vs zip (already lazy). Converting a lazy iterable to a list with list() forces eager evaluation and consumes memory proportional to the entire sequence.

When to use a generator (lazy): When you only need to iterate once, and the sequence is large or expensive to compute.

When to use a list (eager): When you need random access (indexing), multiple passes over the data, or when the dataset is small enough that memory is not a concern.

The rule of thumb: If you can avoid storing the whole dataset in memory, do it. Start with a generator; only switch to a list if you run into a use case that genuinely requires it.

# Quick memory measurement example
import sys

# Generator expression for 10 million squares
lazy_squares = (x * x for x in range(10_000_000))
print(f"Generator object size: {sys.getsizeof(lazy_squares)} bytes")

# Equivalent list (WARNING: don't run this on low-memory machines)
# eager_squares = [x * x for x in range(10_000_000)]
# print(f"List size: {sys.getsizeof(eager_squares)} bytes")

# Practical check: compare memory of list and generator for 1 million items
small_list = [x for x in range(1_000_000)]
small_gen  = (x for x in range(1_000_000))
print(f"List of 1M ints: {sys.getsizeof(small_list) / 1024 / 1024:.1f} MB")
print(f"Generator for 1M ints: {sys.getsizeof(small_gen)} bytes")

The Exhaustion Trap and the iter() Sentinel Form

Here's the behaviour that catches almost everyone at some point: iterators are one-shot. Once an iterator is exhausted, it stays exhausted. Calling iter() on an already-exhausted iterator just returns the same dead object — it does not reset. This is different from calling iter() on an iterable like a list, which creates a brand-new iterator.

This distinction has a practical consequence: if you pass an iterator (not an iterable) to two functions, the second one will silently get nothing. No error. Just an empty loop. These bugs are genuinely hard to track down.

Python also has a lesser-known second form of iter() — iter(callable, sentinel) — which wraps any zero-argument callable into an iterator that keeps calling it until the return value equals the sentinel. This is incredibly useful for reading data in fixed-size blocks, processing queue items, or any situation where you have a 'pull until done' data source.

# ── PART 1: The Exhaustion Trap ──────────────────────────────────────────────

team_members = ["Priya", "Jordan", "Kwame"]  # This is an ITERABLE
team_iterator = iter(team_members)            # This is an ITERATOR (stateful)

print("First loop:")
for member in team_iterator:
    print(f"  Hello, {member}")

print("\nSecond loop over the SAME iterator (iterator is exhausted):")
for member in team_iterator:  # This loop body never executes — silent failure!
    print(f"  Hello again, {member}")

print("  (nothing printed — iterator was already exhausted)")

print("\nSecond loop over the ORIGINAL LIST (always works):")
for member in team_members:  # list is an iterable — fresh iterator each time
    print(f"  Hello again, {member}")


# ── PART 2: The iter(callable, sentinel) Form ─────────────────────────────────

import io

# Simulate reading binary data in 4-byte blocks from a stream
binary_stream = io.BytesIO(b"TheCodeForge.io rocks!")

print("\nReading binary stream in 4-byte blocks:")

# iter(callable, sentinel): calls binary_stream.read(4) repeatedly
# Stops automatically when read() returns b'' (empty bytes — end of stream)
for block in iter(lambda: binary_stream.read(4), b""):
    print(f"  Block: {block}")  # Each block is exactly 4 bytes (except maybe last)

When to Reach for an Iterator Instead of a List (And When to Run Away)

Here's where most devs get it wrong: they use iterators because they heard they're "memory-efficient" without asking if their data actually benefits from laziness. The decision isn't philosophical—it's about access patterns.

Use iterators when you're processing data one element at a time, never needing random access, and the dataset is larger than available RAM. Streaming CSV files, parsing network packets, generating sequences on the fly. These are iterator territory.

Avoid iterators when you need to index into the data, iterate over it multiple times, or modify elements in place during iteration. A list is not your enemy—it's the right tool when you need random access or multiple passes without re-initializing the iterator.

The rule is brutal but simple: if your data fits in memory and you access it more than once, use a list. If your data doesn't fit in memory or you only traverse it once, use an iterator. Don't cargo-cult memory efficiency.

// io.thecodeforge — python tutorial

# Bad: iterator for random access (explodes at runtime)
def parse_log_files(file_paths):
    lines = (line for path in file_paths for line in open(path))
    if lines[42]:  # TypeError! Generator not subscriptable
        print("Nope.")

# Good: iterator for streaming (memory constant)
def stream_recent_logs(file_path):
    with open(file_path) as log:
        for line in log:
            if "ERROR" in line:
                yield line

# Use list when you need index access
log_entries = open("server.log").readlines()
print(log_entries[42])  # Works fine

Creating Different Types of Iterators: Yield Original, Transform, or Generate New Data

Not all iterators are created equal. Once you understand the iterator protocol, you need to know which flavor solves your problem. There are three distinct patterns you'll see in production—and mixing them up causes subtle bugs.

Yielding original data means your iterator doesn't modify the source—it just exposes it lazily. Think reading a file line by line without stripping or parsing. This is the simplest, safest pattern because the consumer controls transformation.

Transforming input data is where most pipeline code lives. Your iterator yields a modified version of each element—parsing raw bytes into structs, converting log timestamps, normalizing text. The key: every element transforms independently.

Generating new data means you're producing values that have no direct mapping to input. An iterator that produces Fibonacci numbers, a counter, or a sliding window over a stream. No external source, just logic.

Each pattern demands different testing strategies. Yielding original data is trivial to unit test. Transforming requires input/output pairs. Generating needs convergence checks to avoid infinite loops.

// io.thecodeforge — python tutorial

def original_data(path):
    """Yields original lines, no transformation."""
    with open(path) as f:
        yield from f

def transform_data(lines):
    """Transforms each line into a tuple."""
    for line in lines:
        parts = line.strip().split(",")
        yield (parts[0], int(parts[1]))

def generate_ids(prefix, count):
    """Generates new data from pure logic."""
    for i in range(count):
        yield f"{prefix}_{i:04d}"

# Usage
lines = original_data("users.csv")
transformed = transform_data(lines)
for uid, count in transformed:
    print(f"{uid}: {count} entries")

for uid in generate_ids("session", 3):
    print(f"Logging {uid}")

Coding Potentially Infinite Iterators — The Pattern That Breaks Beginners

Infinite iterators aren't a gimmick—they're how you model real-time data streams, retry loops, or sensor feeds. But infinite means you never get a StopIteration. If you write a for loop over one, you hang. Forever.

The pattern is simple: write an iterator that never raises StopIteration, and control consumption from the caller side. Use itertools.islice, takewhile, or explicit break conditions. The generator function with yield is your cleanest tool here.

The danger? Forgetting to add a break condition in a production loop. I've seen a batch processing job run for 14 hours because an infinite iterator fed into a for loop with no termination logic. The code looked correct until you traced the data flow.

Best practice: always wrap infinite iterators with a bounded consumer. Either pass a count limit or use itertools.takewhile with a predicate. If someone else uses your iterator, they won't expect it to hang—make the infinite nature explicit in the function name.

// io.thecodeforge — python tutorial

import itertools
import random

def sensor_temperature():
    """Infinite iterator: never raises StopIteration."""
    while True:
        yield random.gauss(25.0, 2.0)

def bounded_take(iterator, count):
    """Safe wrapper: limits consumption."""
    for _ in range(count):
        yield next(iterator)

# Safe usage with islice
for temp in itertools.islice(sensor_temperature(), 100):
    if temp > 30:
        print(f"Alert: {temp:.1f}°C")

# Dangerous: forever loop
# for temp in sensor_temperature():  # Hangs!
#     print(temp)

Stop Writing boilerplate — Subclass collections.abc.Iterator Instead

Every time you hand-roll __iter__ and __next__ on a class, you're writing code that Python already gave you. The collections.abc module ships with Iterator — an abstract base class that automatically provides __iter__ for you. You just implement __next__. That's it.

Why does this matter? Because __iter__ returning self is boilerplate you will forget, and when you forget it, your iterator won't work in for loops. Iterator.__subclasshook__ also catches classes that implement __next__ without explicit inheritance, so your code stays duck-typed friendly.

In production, this pattern matters when you're building streaming data pipelines, file parsers, or any component that processes chunks. Subclassing Iterator signals intent — every dev on your team immediately knows this class is meant to be exhausted. No guesswork, no hidden state bugs.

// io.thecodeforge — python tutorial

from collections.abc import Iterator

class LogTailer(Iterator):
    def __init__(self, path: str):
        self.file = open(path)
    
    def __next__(self) -> str:
        line = self.file.readline()
        if not line:
            self.file.close()
            raise StopIteration
        return line.strip()

// Usage
for entry in LogTailer("/var/log/syslog"):
    if "ERROR" in entry:
        print(entry)
        break

Why You Should Inherit From collections.abc.Iterator (And Not Just Wing It)

Hand-rolled iterators break silently in subtle ways. Your custom class has __next__ but some copy-paste rookie forgets __iter__? Now it fails in list() and for loops with a TypeError: 'YourClass' object is not iterable. That's a 30-minute debugging session where you stare at code that clearly has __next__ and scream at your monitor.

Subclassing Iterator eliminates that entire class of bug. The ABC provides __iter__ returning self, plus mixin methods like __length_hint__ that help CPython optimize memory in list() calls. Your iterator becomes a first-class citizen — it plays nice with itertools, multiprocessing, and any function that expects an iterable.

When you're building production data pipelines, this isn't about elegance — it's about consistency. Every team member follows the same contract. Your code review notes go from 'add __iter__' to 'approved', and you get back to shipping features.

// io.thecodeforge — python tutorial

from collections.abc import Iterator

class ChunkReader(Iterator):
    def __init__(self, data: list, chunk_size: int):
        self.data = data
        self.chunk_size = chunk_size
        self._index = 0

    def __next__(self) -> list:
        if self._index >= len(self.data):
            raise StopIteration
        chunk = self.data[self._index:self._index + self.chunk_size]
        self._index += self.chunk_size
        return chunk

// Works instantly with standard library
from itertools import islice
logs = list(range(100))
for batch in islice(ChunkReader(logs, 10), 3):
    print(len(batch))