Skip to content

chore(platform): release 2.5.4 (guardrails trace tags, 5 s self-check timeout) - #136

Merged
devopssquaddev merged 3 commits into
mainfrom
chore/release-2.5.4
Nov 17, 2026
Merged

devopssquaddev merged 3 commits into
mainfrom
chore/release-2.5.4

Conversation

@devopssquaddev

Copy link
Copy Markdown
Owner

Summary

Release 2.5.4 (PATCH): guardrails self-check fixes found by reading the 2.5.3 traces on the production-shaped lab.

  • Empty self-check answers were timeouts. 27 of 103 self-check model calls (26%) ended at exactly 1.5 s, the timeoutMs default, while gpt-oss answers in 0.9–1.5 s. A check with no answer let the request through, even in block mode.
  • Fix:
    • selfCheck.timeoutMs default goes from 1500 to 5000.
    • New selfCheck.onUnavailable: allow (default, unchanged) or block, which refuses in block mode when a check gets no verdict. It is ignored in log mode.
  • Trace tags:
    • Every check records a guardrail self_check <side> span with zelkor.guardrail.{mode, verdict, status, enforced, on_unavailable, latency_ms}, mirrored as Langfuse guardrail_* metadata.
    • A trace now shows whether a check was log-only or enforcing, what it decided, and whether it timed out.
  • docs/nemo-guardrails.md documents the new key and the trace fields.

Verification on the production-shaped lab

Check Result
Empty self-check answers 0 of 36 (was 26%). Median 0.98 s, max 3.53 s
Trace tags Every check tagged. Harmful prompts in log mode show verdict=block enforced=False
Fail closed: block + onUnavailable: block, timeout forced to 10 ms Benign prompts refused, guardrail_unavailable logged
S-gate, default mode 87 passed. S-12 hit the gateway's per-minute limit (429), as on 2.5.3
Refusal path under block 7 of 7 (S-12, S-13, S-81, UAT-12, drop-in harmful)
S-71/72/73 tool policy 3 of 3
Scenario subset (traces, UAT-24, drop-in, FinServe isolation and sandbox) 28 passed, 0 failed
Offline suites 762 passed
Fail-closed matrix PASS 6, GAP 1 (DNS egress)

🤖 Generated with Claude Code

devopssquaddev and others added 3 commits November 17, 2026 14:00
…nUnavailable

On a live cluster 27 of 103 self-check calls (26%) came back empty: all
at exactly 1.5 s, the timeoutMs default; gpt-oss answers in 0.9-1.5 s.
An empty check let the request through even in block mode, and a trace
did not show the mode, the verdict or whether it was enforced.

- every check records a "guardrail self_check <side>" span with
  zelkor.guardrail.{mode, verdict, status, enforced, on_unavailable,
  latency_ms}, mirrored as Langfuse guardrail_* metadata
- status tells timeout, error and unparseable apart
- selfCheck.timeoutMs default 1500 -> 5000
- selfCheck.onUnavailable: allow (default, unchanged) | block refuses in
  block mode when a check has no verdict; ignored in log mode
- docs: the new key and the trace fields

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… timeout)

- charts, first-party image tags, Zelkor Dockerfile bases and example
  charts at 2.5.4; example chart dependencies rebuilt
- profiles/values-production.yaml at tag 2.5.4, digests empty until the
  signed release run
- THIRD_PARTY_NOTICES: 2.5.4 row (no third-party changes)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S-gate against the 2.5.4 candidate on the production-shaped lab: default
self-check mode (log), the guardrail-refusal cases (S-12, S-13, S-81,
UAT-12, drop-in harmful) under the selfCheck.input: block opt-in, and
S-71/72/73 under a temporary tool-policy overlay. File paths removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devopssquaddev
devopssquaddev merged commit e47c7ea into main Nov 17, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant