Python Queue task_done: completion accounting includes shutdown items

Operation contract

A receipt printer consumes two jobs and then a shutdown sentinel. Every successful get() creates a completion obligation. Calling task_done() in finally satisfies that obligation even when the worker returns on the sentinel. A completed queue counter says all items were acknowledged by workers; it does not prove a remote printer or database committed the result. Idempotent job handling covers replay after a process failure.

Failure and ownership boundary

One worker makes output order stable. With several workers, enqueue one sentinel for each worker after ordinary work, or use an explicit shutdown protocol. Calling task_done() twice raises ValueError. A daemon worker can be terminated at process exit, so it is not a durability guarantee.

Working program

from queue import Queue
from threading import Thread

jobs = Queue()
processed = []

def print_receipts():
    while True:
        receipt_id = jobs.get()
        try:
            if receipt_id is None:
                return
            processed.append(receipt_id)
        finally:
            jobs.task_done()

worker = Thread(target=print_receipts)
worker.start()
for receipt_id in ("R-47", "R-48"):
    jobs.put(receipt_id)
jobs.put(None)
jobs.join()
worker.join()
print("processed:", processed)
print("unfinished:", jobs.unfinished_tasks)

Output

processed: ['R-47', 'R-48']
unfinished: 0

Costs and limits

An unlimited queue retains all submitted payloads. Set maxsize and measure both item count and retained bytes.

Common Mistakes

• A sentinel needs task_done() too.
• Queue.join() is not proof of an external commit.
• Do not submit unbounded payloads to an unlimited queue.

Connected lessons

Python asyncio.Queue: backpressure and completion accounting, Python thread pools: collect results and observe worker failures, Python SQLite job project: reject conflicting replays by request identity.

Continue with Python Queue.shutdown: drain accepted work before workers exit.