Skip to content

[refactor] Extract assertion-evaluation fixes from #49#61

Merged
line-o merged 8 commits into
eXist-db:developfrom
joewiz:feature/assertion-fixes
May 30, 2026
Merged

[refactor] Extract assertion-evaluation fixes from #49#61
line-o merged 8 commits into
eXist-db:developfrom
joewiz:feature/assertion-fixes

Conversation

@joewiz
Copy link
Copy Markdown
Member

@joewiz joewiz commented May 29, 2026

[This response was co-authored with Claude Code. -Joe]

Summary

Small, focused extraction of the assertion-evaluation bug fixes from #49, per @line-o's request to keep this round's changes phone-reviewable and to unblock an RC2 cut. Also folds in the admin-authentication change from #49 (commit 5 below), which is a prerequisite for the in-flight EXPath File built-in extension PR (eXist-db/exist#6257).

8 commits, touching only TestCaseRunnerActor.scala and ExistServer.scala. None overlap with #58. Compile is clean against current develop. XQTS HEAD measured before and after with a companion eXist fix (eXist-db/exist#6409) in place.

What's in scope

# Commit Effect
1 [bugfix] Trim expected XML in assertXml to avoid whitespace mismatches Eliminates false assertXml failures from formatting newlines in catalog CDATA
2 [bugfix] Preserve namespace URI in error code comparison for non-standard namespaces Matches Q{ns}local-form error codes (e.g. EXPath File Q{http://expath.org/ns/file}is-dir) instead of bare local name
3 [bugfix] Treat assertion evaluation errors as failures, not runner errors XPathException raised during assertion eval is now a failure (the assertion failed), not an error (the runner crashed)
4 [refactor] Add raw serialization and serialization properties to ExistServer Supporting refactor for serialization-matches and JSON-output assertions
5 [bugfix] Use admin authentication for embedded server connections Required for tests touching privileged modules (filesystem etc.). Surfaces 11 pass→fail in XQ 3.1 HEAD under admin — see "Admin-auth behavior changes" below.
6 [bugfix] Handle AllOf assertions containing Error in test case evaluation Matches error code against Error inside <all-of> (previously only direct Error and <any-of> were handled)
7 [bugfix] Fix XMLUnit comparison argument order and filter ignorable whitespace Fixes inverted "Expected"/"but was" labels and drops whitespace-only text nodes during XMLUnit diff
8 [bugfix] Pass result as context item for single-item assert evaluations Fixes XPDY0002 for assertions using the ? lookup operator on single-item results (e.g. ?columns on a parse-csv map)

Commits 1–5 (in chronological order: trim-XML, preserve-NS-URI, errors-as-failures, raw-serialization, admin-auth) cherry-pick cleanly from #49. Commits 6–8 (AllOf-with-Error, XMLUnit-fix, context-item) are hand-rewrites against current develop: their original #49 SHAs modify file regions that exist only on the QT4/Update/XQFTTS base branch and don't apply directly. Each hand-rewrite is the assertion-relevant portion of its source commit; the structural QT4/Update bits were dropped.

What was considered but deferred

#49 commit Why deferred
[feature] Implement XQTS sandpit support for writable test directories Structural feature — belongs in the suite-additions wave
[bugfix] Propagate static base URI to assertion evaluation context Depends on sandpit
copy-modify-return capture (XQUF half of 9a316ac) Depends on runUpdateTestCase (XQUF)
normalizeWhitespace() plumbing (XQFTTS half of ea771f6) XQFTTS-specific; pulled the arg-order fix and the whitespace-only filter from the same commit instead
[bugfix] Propagate testCase environment namespaces to assertion queries Already present on develop via #52 (different design — parameter passing instead of actor-private var) — confirmed redundant

Verification

XQTS HEAD measured before and after using the same shaded exist-core. The "after" measurement is the runner PR + the companion eXist fix #6409 in place:

              BEFORE    AFTER    DELTA
tests          26996    26460       
pass           23999    23555      
fail            1788     1723      
error            147      131      
skip            1062     1054
pass-pct       88.90%   89.02%   +0.12pp

Test movement (transitions where both runs have data):

fail->pass            15    e.g. app-FunctxFn/functx-fn-root-all, app-Walmsley/d1e78807c
error->fail           15    e.g. fn-json-to-xml/json-to-xml-008
pass->fail             5    detailed below
error->pass            1    method-xml/K2-Serialization-17
  • The 15 fail→pass are the genuine fixes from commits 1, 6, 7, 8.
  • The 15 error→fail are the intentional reclassification from commit 3.
  • The 5 pass→fail are detailed below; all 5 are accounted for.

Other verification:

  • sbt compile clean (only pre-existing 23 Scala warnings, none introduced)
  • mvn test -pl exist-core on develop unchanged by this PR (the runner is consumed by the exist-xqts appassembler bootstrap, not by exist-core's test JVM); 6,592 tests / 0 failures / 0 errors / 106 skipped, 3:39 wall.

Admin-auth behavior changes (commit 5)

Commit 5 changes the embedded server connection from getBroker() (guest) to authenticate("admin", "") (admin). This is a prerequisite for the in-flight EXPath File built-in PR (#6257) — privileged modules need admin to operate. The change surfaces 11 pass→fail transitions in XQ 3.1 HEAD, in three clusters:

Cluster A — fn:environment-variable is dba-only (4 tests):
fn:environment-variable returns the empty sequence for non-dba users. Tests like fn-environment-variable-005 use a defensive guard:

let $all := fn:available-environment-variables()
return empty($all) or ($all = "QTTEST" and fn:environment-variable("QTTEST") eq "42")

Under guest the guard's first clause is true (env vars hidden, function returns ()), so the test passes regardless of fixtures. Under admin the function returns the real env vars, the guard's first clause is false, and the test correctly reports false because the test fixture (QTTEST=42) isn't set in the runner's environment.

These are spec-correct failures revealing a runner-side test-fixture gap, not eXist behavior changes. Affected tests: fn-available-environment-variables-011, fn-environment-variable-005, -006, -007.

Cluster B — lone-slash predicate with admin context (6 tests):
All six are "lone slash" tests of the form fn:count(.[5 * /]) against a single-element source doc. The cause is an eXist optimizer bug where RootNode doesn't declare CONTEXT_ITEM as a dependency; the predicate gets hoisted out of the iteration context, RootNode.eval falls through to the static-context default, and under admin returns content from /db/system/security/. Diagnosed by instrumenting OpNumeric.eval: rseq contains atomized fields from admin.xml, including two RIPEMD160 password hashes. Fixed by #6409 — once that merges, all 6 of these tests flip back to pass.

Affected tests: prod-PathExpr/PathExpr-1, -2, -15 and prod-StepExpr/Steps-leading-lone-slash-15, -17, -1a.

Cluster C — fn-transform-43 (1 test, unrelated):
fn-transform-43 is an fn:transform test that calls parse-xml on a serialized result. Fails in the batched XQTS run only — passes when running fn-transform alone, or fn-transform-43 in isolation. The diff is cross-test-set JVM state pollution (Saxon configuration cache, XSLT compilation cache, or similar), not anything tied to admin auth or this PR's changes. Predates these changes; not blocking; worth a separate look later.

Net assessment with #6409 applied: Cluster A is correct behavior surfaced (4 tests, document); Cluster B is fully fixed (6 tests recovered); Cluster C is a 1-test pre-existing batched-run intermittency. Keeping commit 5 in this PR unblocks the path to EXPath File integration (#6257) and ships strictly positive on the XQTS measurement once #6409 lands.

Suggested merge order

  1. Wait for #6409 (the RootNode fix) to merge to eXist develop
  2. Wait for exist-core snapshot to publish via the runner's Maven resolution
  3. Merge this PR
  4. Cut RC2 and let eXist's exist-xqts dep-bump follow

Related

joewiz and others added 8 commits May 28, 2026 14:46
Test catalog CDATA often has formatting newlines (e.g., after </result>
before ]]>) that become spurious text nodes inside the ignorable-wrapper
element, causing child count mismatches like "Expected child nodelist
length '1' but was '2'".

Trim the expected XML string before wrapping it in the ignorable-wrapper
element to eliminate these false failures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…dard namespaces

The runner was extracting only the local part of XPathException error
codes, losing the namespace URI. QT4 test catalogs use EQName notation
(e.g., Q{http://expath.org/ns/file}is-dir) for extension module error
codes. Now error codes from non-standard namespaces (not xqt-errors or
exist-xqt-errors) are formatted as Q{ns}local for proper matching.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rors

When an XPath assertion (assert, assert-eq, assert-deep-eq,
assert-permutation, assert-serialization) raises a query error
during evaluation (e.g., XPTY0004 type mismatch), this indicates
the assertion failed — not that the runner itself errored.

Previously these were reported as ErrorResults, inflating the error
count and masking the real nature of the failure. Now they are
reported as FailureResults with the error details in the message.

Fixes fn-parse-json-717 and fn-parse-json-731 which errored due to
type mismatches in assertion XPath evaluation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tServer

Add sequenceToStringRaw() for serialization-matches assertions where
the exact serialized output must be preserved without newline
replacement. Refactor sequenceToString into a shared implementation
with a sanitize flag.

Add serializationProperties field to Result to capture query context
serialization options (e.g., declare option output:method "json") for
use in assertion evaluation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change getBroker() to authenticate("admin", "") to get an admin-level
broker instead of a guest broker. Required for modules that need
filesystem or privileged access (e.g., EXPath File module operations).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tion

When a query returns an error and the expected result is an AllOf
assertion containing an Error assertion, match the error code against
the Error inside the AllOf. Previously, only direct Error and AnyOf
assertions were matched; AllOf was not handled, causing false failures
for tests like EXPath File read-binary bounds checking where the
catalog wraps the expected error in <all-of>.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…hitespace

Fix two related issues in findDifferences (used by assertXml):

1. The DiffBuilder.compare() and withTest() arguments were swapped,
   causing 'Expected' and 'but was' labels in error messages to be
   reversed (the actual was treated as the expected and vice versa).

2. Add a node filter that drops whitespace-only text nodes before
   comparison, so insignificant whitespace in test catalog CDATA
   does not produce spurious child-count or text-mismatch failures.

Combined extraction of the assertion-correctness portions of
upstream commits 9a316ac and ea771f6. The normalizeWhitespace plumbing
from ea771f6 (specific to XQFTTS) is deferred to the suite-additions
wave; this commit keeps the runner XQ 3.1-only and surgical.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
XQTS assert expressions like ?columns and ?get(1,1) use unary lookup
which requires a context item. Previously the result was only available
as the $result variable but not as the context item, causing XPDY0002
errors for any assertion using the ? lookup operator.

Set context only for single-item results (e.g., maps from parse-csv)
to avoid per-item evaluation for multi-item sequences such as
csv-to-arrays output.

Extracted from upstream commit c6f3caa; assertion-only.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@line-o line-o self-requested a review May 30, 2026 11:19
@line-o line-o self-assigned this May 30, 2026
Copy link
Copy Markdown
Member

@line-o line-o left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pulling this in and cutting a RC2 afterwards

@line-o line-o merged commit 75cbde1 into eXist-db:develop May 30, 2026
6 checks passed
@joewiz joewiz deleted the feature/assertion-fixes branch May 30, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants