Repository navigation
tests: check the extras gate's verdict against the environment - #75
Merged
Merged
Conversation
The two module floors cannot see a stuck verdict. With the gate forced to "absent" in an environment that does have the server extra, _collect skips the subtree, `skipped` receives all five names from _server_subtree() either way, so the server floor is met by construction, and every example lives in a core module so the example floor is met too. Both floors stay green while the five server modules' examples quietly stop running. A count that a correct skip also produces cannot separate a correct verdict from a stuck one; only importing the subtree and comparing against the verdict can. test_server_gate_matches_the_environment does that, with three assertions: the verdict agrees with the import in both directions; where the verdict is "absent" and the import does fail, the missing root is one _SERVER_REQUIRES names; and a Skipped escaping the import is caught by hand, because pytest reports an escaped skip as a skip and not a failure. Measured on this tree on 2026-11-09. Forcing the verdict can only redden where the verdict is wrong, so two of the four rows have nothing to catch, and that is the expected result rather than a gap: - env with the extra, verdict forced absent: 1 failed, 1 passed - "the gate says the extra is absent, but the whole subtree imported". The doctest test stays green, which is the defect this commit exists for. - env with the extra, verdict forced present (correct): 2 passed, nothing to catch. - no-extras wheel env, real verdict absent (correct): 2 passed, nothing to catch. - no-extras wheel env, verdict forced present: both tests red - "importing the subtree failed on 'fastapi'". The other two assertions, each with the mutation's anchor printed: - unnamed root: a wheel built with `import psimodpy_unnamed_root` ahead of the fastapi import in server/__init__.py, run with no extras and the real verdict, gives 1 failed, 1 passed - "the import does fail, but on 'psimodpy_unnamed_root', which _SERVER_REQUIRES does not name". The skip was right by accident; the doctest test is green either way. This is mzmlpy's base-install defect in the shape where it is silent. - escaped Skipped: `pytest.skip(...)` at import in server/__init__.py gives 1 failed, 1 skipped with the hand-catch, and 2 skipped at exit 0 with the except clause removed - the whole file green while the gate is dead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The first version of this test read the missing root off whichever name under
psimodpy.server failed first. That is too weak: a half-installed extra is a
supported environment, and the modules in a gated subtree need not fail on the
same root, or at all. `pip install fastapi pydantic` without mcp is enough to
make the subtree unimportable, and tdfpy measured the shape where only the one
module that actually touches the missing dependency fails.
So the predicate is now two parts, over a dict of {module: missing root}: every
failure must name a root _SERVER_REQUIRES lists, and at least one failure must
exist when the verdict says absent. It is deliberately not "every gated module
fails" - measured here with a wheel whose server/__init__.py does not import
server.app, and no extras installed: 3 of the 5 names fail (app on fastapi,
models and references on pydantic) while psimodpy.server and server.dashboard
import fine. The old shape and "every module fails" would both have called that
environment broken.
Measured failing directions, on this tree, each with its mutation anchor printed:
- every failure names a listed root: a wheel importing psimodpy_unnamed_root
ahead of fastapi, no extras -> 1 failed, "failed to import on a root
_SERVER_REQUIRES does not name", naming all five modules and that root.
- verdict absent but nothing fails: extras present, _installed forced False ->
1 failed, "the gate says the psimodpy.server extra is absent, but every name
under it imported", listing the five.
- verdict present but something fails: no extras, _installed forced True ->
1 failed, "the gate says the extra is present, but these modules failed",
all five on fastapi.
- the two correct-verdict rows have nothing to catch and stay at 2 passed: a
forced verdict can only redden the environment it is wrong about.
- the Skipped hand-catch: a pytest.skip injected into _subtree_failures gives
1 failed, "importing the server subtree raised Skipped, which pytest would
report as a skip"; with the except branch deleted the same injection gives
1 skipped at exit 0, which is the failure mode the hand-catch exists for.
Half-installed cross-check: wheel plus fastapi and pydantic, no mcp -> verdict
absent, all five names fail on 'mcp', 2 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tests/test_doctests.pyskips thepsimodpy.serversubtree whenfastapi,mcporpydanticis missing. Nothing checked that skip's verdict against the environment, and no floor above it can: with the verdict stuck at "absent" where the extra is in fact installed,_collectskips,skippedreceives all five names from_server_subtree()either way, so the server floor is met by construction, and every>>>example lives in a core module so the example floor is met too. Both floors stay green while five modules' examples quietly stop running. A count that a correct skip also produces cannot tell a correct verdict from a stuck one. This test imports the subtree and compares.The predicate is deliberately not "every gated module fails when the verdict is absent". A half-installed extra is a supported environment, and the modules in a gated subtree need not fail on the same root, or at all. So it is two parts over a dict of
{module: missing root}: every failure must name a root_SERVER_REQUIRESlists, and at least one failure must exist when the verdict says absent. Measured with a wheel whoseserver/__init__.pydoes not importserver.app, no extras installed: 3 of the 5 names fail (server.apponfastapi,server.modelsandserver.referencesonpydantic) whilepsimodpy.serverandserver.dashboardimport fine. "Every module fails" would have called that environment broken. Half-installed cross-check, wheel plusfastapiandpydanticwithoutmcp: verdict absent, all five fail onmcp, 2 passed.Each of the three assertions has a measured failing direction, every mutation applied with its anchor printed beside the result:
psimodpy_unnamed_rootahead offastapi, no extras_SERVER_REQUIRESdoes not name", naming all five modules and that root_installed→False)_installed→True)fastapi(the doctest walk fails too, so the file is 2 failed)Skippedpytest.skipinjected into_subtree_failures: 1 failed, "importing the server subtree raised Skipped, which pytest would report as a skip". With theexceptbranch deleted, the same injection is 1 skipped at exit 0 — the failure mode the hand-catch exists forTwo of the four forced-verdict rows have nothing to catch, and that is the shape of the thing rather than a gap in the measurement: a forced verdict can only redden the environment it is wrong about, so forcing "present" where the extra is present, or "absent" where it is absent, changes no observable. Saying so beats claiming four reddenings.
One thing to expect on this PR: the
elsebranch only runs where the extra is absent, and CI installs it everywhere, so no job covers that branch.[tool.coverage.run] source = ["src/psimodpy"]means test files are not measured here andcodecov/patchcannot see it, but read the row count and the conclusion bucket together rather than the summary, and name any non-pass row.just check psimodpyfrom the workspace root: ajs 2452, exit 0, 13.2 s, normal 1-cpu job. No version bump, no CHANGELOG heading, no tag, no other package, and the gate itself is unchanged.🤖 Generated with Claude Code
https://claude.ai/code/session_01Jzz8tTh1pMWBFrRYgqBkWB