Python InterpreterPoolExecutor: pass results across isolated runtimes

Last updated: 1 Oct 20265 min read

tutorial

Interpreter workers have separate module state; return values instead of expecting a shared mutable global.

Operation contract

Two receipt amounts are processed by isolated interpreters. Each worker appends to its own module-level list, returns a score, and the parent collects ordered results. The parent's list remains empty. Run this source as a file so the top-level callable can be found by workers.

Failure boundary

The executor serializes submitted callables and arguments. Large payloads may spend more time crossing the boundary than the calculation saves. Extension modules must support subinterpreters, and a worker-local global is not shared state or persistent storage. Do not use this fixture as evidence of a multi-core speedup.

Working program

from concurrent.futures import InterpreterPoolExecutor

reviewed_amounts = []

def score_receipt(amount):
    reviewed_amounts.append(amount)
    return amount * 2

if __name__ == "__main__":
    with InterpreterPoolExecutor(max_workers=2) as pool:
        scores = list(pool.map(score_receipt, (47, 48)))
    print("scores", scores)
    print("parent_reviews", len(reviewed_amounts))

Output

scores [94, 96]
parent_reviews 0

Costs and limits

Each worker owns interpreter state and incurs startup plus serialization overhead. Parallel CPU work needs enough duration to repay those costs.

Common Mistakes

A module global in one interpreter is not the parent's global.
A local closure or unsupported extension may fail to cross the worker boundary.
A returned value is not a durable write to a database.

Connected lessons

Test this contract.

python

interpreter-pool-isolation