Pandas categorical values: reject unknown labels before conversion

Operation contract

The receipt-state fixture declares pending, accepted and rejected in workflow order. It validates each received state against that set before conversion. An unchecked unknown label becomes missing during conversion, which would conceal a rejected input if missing and unknown were treated alike. Sorting the accepted categorical values follows the declared category order rather than alphabetic order.

Failure and ownership boundary

Category order does not itself enforce valid transitions between states. A missing source value also needs a separate policy, and categorical codes are representation details rather than stable wire IDs. Memory savings depend on cardinality and representation; this fixture reports behavior, not a measured universal ratio. Python Enum: parse wire values into a closed state set and Python aggregation exercise: validate records before returning group totals explain other state boundaries.

Tested environment

Dependency check: this program was executed on CPython 3.14.6 with pandas==3.0.6. Install these versions in a separate virtual environment. The download includes the recorded environment snapshot; no third-party package is part of the website runtime.

Working program

import pandas as pd

states = ["pending", "accepted", "rejected"]
dtype = pd.CategoricalDtype(states, ordered=True)
received = pd.Series(["accepted", "pending", "rejected"], dtype="string")
if not received.isin(states).all(): raise ValueError("unknown state rejected")
accepted = received.astype(dtype)
print("workflow order:", accepted.sort_values().astype(str).tolist())
unchecked = pd.Series(["archived"], dtype="string").astype(dtype)
print("unchecked unknown becomes missing:", bool(unchecked.isna().iloc[0]))
print("explicit validation:", bool(pd.Series(["archived"]).isin(states).all()))

Output

workflow order: ['pending', 'accepted', 'rejected']
unchecked unknown becomes missing: True
explicit validation: False

Costs and limits

Validation scans n source values; categorical conversion retains codes and a category dictionary. High-cardinality data can erase expected memory benefits. Category transitions and versioned schema migration remain application work.

Common Mistakes

• Validate unknown labels before they are converted to missing values.
• Do not publish categorical codes as stable business identifiers.

Connected lessons

Python Enum: parse wire values into a closed state set, Pandas nullable integers: missing amounts are not zero, Python aggregation exercise: validate records before returning group totals.