Python itertools.tee: a lagging reader retains buffered values

Operation contract

Two receipt readers share a three-item source. The fast reader consumes two IDs before the slow reader consumes one; the slow reader can still retrieve the first ID because tee retained it. The fixture then drains both. This is not two independent upstream queries or a free clone of a generator.

Failure and ownership boundary

If one reader runs far ahead or never resumes, the buffer can grow with the entire unread stream. tee iterators are not safe for simultaneous use from multiple threads. When full replay is needed and the input is already small and bounded, materializing a list can make the memory cost easier to reason about. Python iterators: exhaustion and repeatable collection ownership, Python itertools: adjacent groups and shared iterator consumption and Python SpooledTemporaryFile: rollover is a storage choice, not an upload limit show other choices.

Working program

from itertools import tee

source_reads = []
def receipt_stream():
    for identifier in ("R41", "R42", "R43"):
        source_reads.append(identifier)
        yield identifier

fast, slow = tee(receipt_stream())
print(next(fast), next(fast))
print("upstream:", source_reads)
print("slow:", next(slow))
print("fast rest:", list(fast))
print("slow rest:", list(slow))

Output

R41 R42
upstream: ['R41', 'R42']
slow: R41
fast rest: ['R43']
slow rest: ['R42', 'R43']

Costs and limits

Total upstream reads are O(n), but retained buffer size depends on the maximum reader lag and can reach O(n). The three-item fixture does not measure a long-running service.

Common Mistakes

• tee can retain a large unread backlog.
• Two tee readers must not be consumed concurrently from different threads.

Connected lessons

Python iterators: exhaustion and repeatable collection ownership, Python itertools: adjacent groups and shared iterator consumption, Python SpooledTemporaryFile: rollover is a storage choice, not an upload limit.