Python Zstandard decompression: stop at an output budget

Last updated: 1 Oct 20265 min read

tutorial

A compressed frame can expand far beyond its input size; bound decoded bytes while consuming it.

Operation contract

A fixed 47,000-byte receipt payload is compressed locally. The incremental decompressor is asked for at most 4,097 bytes, one past a 4,096-byte intake budget. It reports that the frame is over budget without allocating the complete decoded payload through the one-shot decompress function.

Failure boundary

This program supplies the full compressed frame in memory, so production intake still needs compressed-byte and time limits. If the output is within budget, finish decoding and require end-of-frame, then decide how trailing data or multiple frames are handled. The one-shot function can allocate the full decoded result and is unsuitable as the budget check.

Working program

from compression import zstd

receipt_payload = b"R" * 47000
frame = zstd.compress(receipt_payload)
decoder = zstd.ZstdDecompressor()
prefix = decoder.decompress(frame, max_length=4097)
print("over_budget", len(prefix) > 4096)
print("complete_frame", decoder.eof)

Output

over_budget True
complete_frame False

Costs and limits

The decision retains O(limit) decoded bytes here, plus the compressed frame and decoder state. Decompression CPU and memory still need limits for hostile inputs.

Common Mistakes

Compressed size does not cap decoded size.
A partial decode is not a successful integrity check of the full frame.
Bound compressed input, decoded output, and elapsed time separately.

Connected lessons

Test this contract.

python

zstd-decompression-budget