Skip to content

chore(platform): release 2.5.5 (trace error levels, guardrail fields on the model call, CLI fixes) - #138

Merged
devopssquaddev merged 6 commits into
mainfrom
fix/platform-trace-error-levels
Nov 17, 2026
Merged

devopssquaddev merged 6 commits into
mainfrom
fix/platform-trace-error-levels

Conversation

@devopssquaddev

Copy link
Copy Markdown
Owner

Summary

Release 2.5.5 (PATCH): fixes from the trace review of the 2.5.4 gate run, plus two zelkor CLI bugs.

  • Failures that looked healthy are now errors (via OTel span status, as the uniform trace contract requires):
    • a self-check with no verdict (timeout, error, unparseable) makes its model call and the guardrail span ERROR, e.g. self-check timeout; request allowed;
    • a 4xx answer from NeMo is an ERROR span (HTTP 429).
  • Guardrail result where people look: zelkor.guardrail.* / Langfuse guardrail_* metadata is now on NeMo's self_check_<side> <model> call, and per side (guardrail_input_mode, …) on the NeMo request span. The 2.5.4 guardrail self_check <side> span stays.
  • Sandbox: an execute result reports status: "denied" (not "error") when the sandbox refused a probe and the script carried on.
  • selfCheck.timeoutMs default 5000 → 15000. FinServe quant checks sometimes ran past 5 s (the reasoning model thinks long); at 15 s every check finished in about 2 s.
  • Tests: trace hygiene treats a log-mode self-check timeout or unparseable answer as a known error (it stays ERROR in Langfuse).
  • CLI:
    • --env now works after the command (zelkor status --env staging, as the docs show);
    • zelkor verify reports a cosign timeout as FAIL instead of a traceback.

Verification on the production-shaped lab

Check Result
Guardrail fields on model call and request span Present for allow and block verdicts (log mode)
Forced 10 ms timeout Model call and guardrail span ERROR, self-check timeout; request allowed
NeMo 4xx (invalid body) Request span ERROR, HTTP 422
Sandbox mknod probe status: denied; benign code success
S-gate (evidence release-evidence/v2.5.5-ce) 103 tests, 0 failures, 11 skipped
Scenario subset at 15 s Agent traces, UAT-24, drop-in, FinServe isolation + sandbox, Qdrant: pass (5 environment skips)
Offline suites 1185 passed. The failures that remain locally are the laptop toolchain (Rust for a fresh venv, Node 6); the live SDK/ingress tests pass against the lab

🤖 Generated with Claude Code

devopssquaddev and others added 6 commits November 17, 2026 15:46
…rail result on the model call; sandbox denied status

A trace review of the 2.5.4 gate run found failures that Langfuse showed at
the normal level, and guardrail fields that were hard to find.

- a self-check with no verdict (timeout, error, unparseable) makes the
  check's model call and the guardrail span ERROR spans with a message
  such as "self-check timeout; request allowed", plus error.type
- the check result (zelkor.guardrail.*, Langfuse guardrail_* metadata) is
  on NeMo's self_check_<side> model call and, per side, on the NeMo
  request span; the separate guardrail span stays
- a 4xx answer from NeMo is an ERROR span ("HTTP 429"); OTel left 4xx
  server spans unset
- sandbox execute results report status "denied" when the sandbox refused
  a probe and the script carried on, instead of "error"; the Deep Agents
  backend still maps it to exit code 1

Level comes from the OTel span status, as the uniform trace contract
requires; no langfuse.observation.level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k timeouts are known trace errors

On the production-shaped lab 3 of 8 FinServe quant self-checks stopped at
5 s while the same checks finished in about 2 s under a 15 s limit (the
reasoning model sometimes thinks long). Now that a timed-out check is an
ERROR span, a 5 s default would fail runs that are otherwise healthy.

- selfCheck.timeoutMs default 5000 -> 15000
- trace hygiene in tests skips a log-mode self-check timeout (the
  request went through); it stays an ERROR in Langfuse. Block mode and
  other statuses still count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r too

A FinServe quant check ran 6.9 s and returned an empty answer (the
reasoning model spent its budget thinking); in log mode the request went
through, as with a timeout. A failed check call (error) still counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a cosign timeout as FAIL

- `zelkor status --env staging` (the documented form, six places in
  docs) failed with "unrecognized arguments"; every subcommand now takes
  --env, overriding the global flag
- `zelkor verify`: a cosign call that hits the 120 s limit is a FAIL line
  with "timed out after 120s" and the summary still prints, instead of a
  traceback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on the model call, CLI fixes)

- charts, first-party image tags, Zelkor Dockerfile bases and example
  charts at 2.5.5; example chart dependencies rebuilt
- profiles/values-production.yaml at tag 2.5.5, digests empty until the
  signed release run
- THIRD_PARTY_NOTICES: 2.5.5 row (no third-party changes)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S-gate against the 2.5.5 candidate on the production-shaped lab: default
self-check mode (log), the guardrail-refusal cases (S-12, S-13, S-81,
UAT-12, drop-in harmful) under the selfCheck.input: block opt-in,
S-71/72/73 under a temporary tool-policy overlay, and the Qdrant tenant
tests re-run after a test-env label fix. 103 tests, 0 failures, 11
skipped. File paths removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devopssquaddev
devopssquaddev merged commit e7ec2e4 into main Nov 17, 2026
11 checks passed
@devopssquaddev
devopssquaddev deleted the fix/platform-trace-error-levels branch November 17, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant