Thread-safety: forbid_randomness mutates process-global numpy.random, causing intermittent false NonDeterministicFormulaError under concurrent simulations

[anth-volk]

Summary

forbid_randomness (added in #500, policyengine_core/simulations/randomness_guard.py) enforces a per-formula invariant by mutating process-global state: it setattrs every public callable of numpy.random and the stdlib random module to a raiser while a formula runs, and it tracks re-entrancy with module-global, non-thread-local counters and no lock:

# randomness_guard.py
_depth = 0                        # module global
_variable_stack: list[str] = []   # module global

def _install():  setattr(module, name, raisers[name])   # mutates numpy.random for the WHOLE process
def _restore():  setattr(module, name, original)

class forbid_randomness:
    def __enter__(self):
        global _depth
        if _depth == 0: _install()
        _depth += 1
        _variable_stack.append(self.variable_name)
    def __exit__(self, *exc):
        global _depth
        _variable_stack.pop()
        _depth -= 1
        if _depth == 0: _restore()

The guard is installed around the formula call in Simulation._run_formula (simulation.py:1111). Because the swap targets the shared numpy.random/random module objects, any thread in the process sees it — including threads that are not running a formula.

This is safe for single-threaded use and for cross-OS-process fan-out (separate memory), but incorrect under intra-process concurrency: threaded web workers, async request handlers, or Modal containers serving concurrent inputs, where multiple simulations are created and computed in one process at once.

Failure modes

Let thread A be inside a formula (guard installed) and thread B be doing anything else in the same process:

1. False positive on legitimate setup randomness. Simulation.__init__ calls np.random.seed(0) (simulation.py:162; also axes seeding :241, dataset sampling :1449/:1665-1667/:1674) — legitimate non-formula randomness that Forbid randomness inside variable formulas #500 explicitly kept. If thread B constructs a simulation while A is mid-formula, B's np.random.seed(0) hits A's raiser and throws NonDeterministicFormulaError, even though B never ran a formula. This is the "the slower concurrent sim finds np.random swapped out from under it" symptom.

2. Misattributed error. The raiser names _variable_stack[-1], which is a process-global stack. So the error blames whatever formula happens to be on top of the shared stack — typically an innocent variable that never touched randomness.

3. False negative / early restore. _depth is shared and unguarded. If A and B are both in formulas and B's __exit__ drives _depth to 0 while A is still running, _restore() fires and un-patches the module mid-formula, so genuine formula randomness in A goes undetected. Interleaved +=/-= on _depth can also desync (early restore, or never-restore).

Field evidence

• cliff-watch#38 (2026-07-03): "Intermittent 'rules-engine formulas must be deterministic' errors from /api/series." The guard blamed slcsp_age_0 and age_head — formulas that do not call randomness — and the failures were intermittent and not reproducible on demand. Both are hallmarks of failure modes (1)+(2): a real seed call elsewhere (setup or another request) tripping the shared guard, misattributed to the stack-top formula.
• policyengine-household-api#1575 / #1576 (2026-06-25): a deploy had to pin policyengine-core<3.26.7 after the guard raised on is_ssi_recipient_for_medicaid during concurrent deployed customer-input tests.

Scope

Intra-process concurrency only (threads / async / concurrent container inputs). Separate OS processes (pytest-xdist, one-sim-per-container fan-out) do not share the module and are unaffected.

Directions (not prescribing here)

1. Thread-local re-entrancy state — make _depth/_variable_stack threading.local(). Necessary but not sufficient: _install/_restore still mutate the shared module, so thread B's setup seed is still exposed to thread A's installed patch.
2. Stop mutating the shared module. Options: install a permanent, thread-aware shim once at import that consults a thread-local "in-formula" flag (no per-formula install/restore); or move enforcement to a static check at variable registration (inspect formula bytecode/globals for numpy.random/random references) so bad formulas fail fast and deterministically at load with zero runtime module mutation.
3. Combination: static gate as the primary, deterministic enforcement + optional thread-safe runtime shim.

Separate from the model-side cleanup (removing np.random from policyengine-us formulas) and the service pins; this issue is specifically the guard's concurrency-safety defect. Related: policyengine-us#8753 (stale-uv.lock CI gap that let this reach production).