perf(pypangraph): decode graphs into typed msgspec models

[ivan-aksamentov]

⚠️ AI-generated contribution. This pull request was implemented by an AI agent. Review the code, tests, and benchmark methodology before merging.

Alternative PRs (mutually exclusive, merge one):

• jsonschema-rs #201: swap the validator, keep dicts; smallest change
• -> msgspec #202 (you are here): typed decode; fastest
• pydantic #203: typed models; most common library

Problem

Pangraph.from_json validates every loaded graph against the JSON schema generated from the Rust types before building the object. On real graphs that validation, not parsing, dominates load time. Issue #200 reports the slowdown.

On the benchmark graph below, parsing the JSON takes about 40 ms while the pure-Python jsonschema pass takes about 2 seconds: the validator walks every node, edit and position in interpreted Python.

This change

Decode the JSON bytes directly into the typed model in pypangraph/model.py with msgspec. Parsing and validation happen together in compiled code, with no intermediate dict tree, so the load is bounded by decompression rather than validation. The models carry the schema's constraints (non-negative integers, single-character alt, the strand enum), and the path, block and node collections read the typed structs. Pangraph still accepts a plain dict, which is validated and converted with msgspec.convert, so an in-memory dict enforces the same schema as a file.

Why this is the one to merge

• Fastest of every option. The load drops from about 2 seconds to about 17 ms, a 117x end-to-end speedup. It is faster than merely parsing the JSON into a dict, because decoding goes straight into compact structs and skips building the dict tree.
• Typed internals, not just a faster check. The loader now works with real types instead of nested dicts. Attribute access replaces string keys, so a wrong field name is a static error and editors autocomplete the model. The other engines that keep dicts cannot offer this.
• Validation and shape are one thing. There is no separate "parse, then validate" seam to keep in sync; the type definition is the schema, and it is enforced on every decode.
• Small, contained model layer. The types live in one file generated to mirror the Rust-derived schema, and only the parse boundary reads them. The public API of Pangraph and its collections is unchanged.

If the goal is the fastest load and better-typed internals, this is the strongest choice.

Benchmark

Fixture: packages/pypangraph/tests/data/staph.json.gz (664 blocks, 6817 nodes, 15 paths; 1.81 MB compressed, 9.72 MB decoded). Median of 7 runs on one machine in the project Python container, measured by packages/pypangraph/benchmarks/bench_load. Correctness parity was verified for every engine: each accepts the valid graph and rejects missing-field, wrong-type, negative-value and bad-strand mutations.

Full load (parse and validate together):

loader	parse + validate	speedup
baseline (json + jsonschema)	1989 ms	1x
jsonschema-rs	47 ms	42x
msgspec (this PR)	17 ms	117x
pydantic	92 ms	22x

Per-phase breakdown. Decompression is shared; this PR replaces JSON parsing and validation with a single typed decode.

Shared (unchanged by this PR):

phase	ms
gzip decompress	23

Parse and validate (what this PR changes):

step	ms
json parse + jsonschema validate (baseline)	1988
msgspec decode + validate (this PR)	17

Methodology notes:

• Precompiling the pure-Python jsonschema validator does not help; the cost is the interpreted traversal, not validator construction.
• fastjsonschema was rejected: it errors on the schema's format: uint annotation.
• format: uint is a decorative annotation; the non-negative range is enforced by minimum: 0. No engine asserts on the format string.

Conclusion: validation is about 98% of load time, and every candidate removes it. msgspec is the fastest because it folds parsing and validation into a single compiled decode that emits typed structs, so the load is bounded by decompression rather than validation.

The three alternatives

All three PRs branch from feat/merge and rewrite the same loader; they cannot be combined.

• jsonschema-rs. Swap the validator, keep dicts. One-line loader change, no downstream impact, about 42x. The minimal, lowest-risk fix; keeps the untyped dict data model.
• msgspec (this PR). Decode JSON bytes straight into typed structs; validation is part of decoding. Fastest overall and gives typed internals, at the cost of a small typed-model layer read by the collections.
• pydantic. Parse and validate into typed models with model_validate_json. Same typed internals from the most widely used validation library, about 22x, slower than msgspec.

Pick msgspec for the fastest load with typed internals; pick jsonschema-rs for the smallest diff, or pydantic to lean on the more common ecosystem.

Work items

• Add typed msgspec models mirroring the schema in pypangraph/model.py.
• Decode graphs into the models in pypangraph/class_graph.py; accept a dict via msgspec.convert.
• Read the typed models in the path, block, alignment and node collections.
• Declare msgspec in pyproject.toml, requirements.txt, and the Python container.
• Add parameterized tests locking in the accept/reject contract.
• Add benchmarks/bench_load and a graph-loading doc.

Possible improvements

• Generate the models from the schema with datamodel-codegen (as the Makefile already does for the dataclass example) so they cannot drift from the Rust types.

Verify

./dev/docker/python bash -c 'pip install -e packages/pypangraph pytest && cd packages/pypangraph && python3 -m pytest -q'
./dev/docker/python bash -c 'cd packages/pypangraph && python3 benchmarks/bench_load'