Python QueueListener: stop the logging worker after queued records drain

Last updated: 1 Oct 20265 min read

tutorial

QueueHandler moves log handling to a listener thread, and the owner must stop that listener before exit.

Observed contract

A receipt logger sends two records into an owned SimpleQueue. A recording handler receives them on the listener thread. The owner stops the listener before reading the output, giving the two messages a defined completion point.

Boundary

SimpleQueue is unbounded. If producers outpace a slow handler, memory can grow without limit. A bounded queue needs an explicit policy for put failures and enough room for the stop sentinel; the example does not implement that service-level policy.

Executable case

import logging
from logging.handlers import QueueHandler, QueueListener
from queue import SimpleQueue

class ReceiptCollector(logging.Handler):
    def __init__(self):
        super().__init__()
        self.records = []

    def emit(self, record):
        self.records.append(record.getMessage())

receipt_queue = SimpleQueue()
collector = ReceiptCollector()
queue_handler = QueueHandler(receipt_queue)
logger = logging.getLogger("aitrove.receipt.queue.contract")
logger.setLevel(logging.INFO)
logger.propagate = False
logger.addHandler(queue_handler)
listener = QueueListener(receipt_queue, collector)
listener.start()
try:
    logger.info("receipt %s accepted", "R-47")
    logger.info("receipt %s exported", "R-48")
finally:
    listener.stop()
    logger.removeHandler(queue_handler)
    queue_handler.close()
    collector.close()

print("delivered", collector.records)

Output

delivered ['receipt R-47 accepted', 'receipt R-48 exported']

Cost

Enqueueing reduces producer time spent in slow handlers, but adds a worker thread, queue memory, and shutdown work. The listener still processes one handler call at a time in this fixture.

Common Mistakes

Stopping the process without listener.stop can leave records unhandled.
An unbounded queue is not a backpressure strategy.
Keep a handler that writes to a shared sink thread-safe or give it one clear owner.

Connected lessons

Test this contract.

python

queue-listener-shutdown