feat(harness): deterministic resolve → status display → runtime wiring (#2693)#4293
Open
SamhandsomeLee wants to merge 9 commits into
Open
feat(harness): deterministic resolve → status display → runtime wiring (#2693)#4293SamhandsomeLee wants to merge 9 commits into
SamhandsomeLee wants to merge 9 commits into
Conversation
The compaction trigger was computed as a percentage of the raw context window in every live path, ignoring the output tokens reserved for the turn. The window is the wrong denominator: input can only ever occupy `window - reserved_output - headroom` (the input budget ceiling), so a percentage of the window can sit above the room input can actually use. Symptoms this fixes: - Tight self-hosted windows (e.g. a 256K vLLM deployment reserving a large output cap) put the 80% trigger above the usable ceiling, so automatic compaction fired too late - the session ran until the provider hard-rejected on context length. - Even V4's 1M window reserves 262K for interleaved thinking, so the old 80%-of-window trigger (800K) sat above the ~736K input can hold. - `ContextBudget` already computed the correct ceiling but then derived its `compaction_trigger_tokens` from the window, and that field was never wired to the live trigger at all - the live thresholds lived in three separate `percent * window` sites instead. Changes: - `ContextBudget` is now the single source of the trigger formula: it exposes `input_budget_ceiling` and `compaction_trigger_for_percent`, and its default trigger field is anchored to the ceiling. Because the percentage is applied to the ceiling (and clamped to <=100%), the trigger can never exceed the ceiling by construction. - `route_budget`'s route and model trigger helpers build a `ContextBudget` and read the ceiling-anchored trigger; no caller computes its own `percent * window` anymore. - The TUI pre-send auto-compact gate compares estimated context against the shared `compact_threshold` (ceiling-derived) instead of a window-relative percentage, keeping it in the same dimension as the engine auto-compaction. - Moved the output-reservation primitives (`TURN_MAX_OUTPUT_TOKENS`, `effective_max_output_tokens`, `route_output_reservation_for_window`) from `core/engine/context.rs` down to `route_budget` so the trigger helpers can reserve output without an upward dependency into the engine; the engine re-exports `effective_max_output_tokens`. - Moved `compaction_threshold_for_model_at_percent` from `models` to `route_budget` for the same reason; `models` stays dependency-free so the `#[path]`-included integration harness still compiles. Behavior change (this is the bugfix, not a regression): with output reserved before measuring fullness, the Default trigger moves earlier on every route. At 80%, deepseek-v4-pro drops 800_000 -> 589_466 and a 256K trinity route drops 209_715 -> 156_467. Auto-compact enable/disable defaults are unchanged. Tests: a deterministic property test sweeps a wide grid of window/output-cap/input/percent and asserts trigger <= ceiling, making the overflow impossible by construction rather than clamped away. Migration tests cover the large-window (V4) and tight-window cases, and the existing engine input-budget tests confirm the reservation math is preserved.
Make HarnessProfile resolution order-independent with provenance and a Standard fallback. Adds resolve_harness, which ranks user profiles ahead of built-in seeds by model_pattern specificity. Pure config logic only; no runtime behavior change (Hmbown#2693).
Add /harness command and a footer posture chip that show the resolved harness posture, its source, and repo-law path separately. Consumes resolve_harness from PR1; preview-only with no runtime behavior change (Hmbown#2693).
Cache resolved HarnessResolution on App and inject posture into EngineConfig for TUI, exec, and runtime-thread paths. Zero runtime consumption until PR3a; adds refresh hooks, Default on HarnessResolution, and tests.
PR3a) PrefixCache compacts later via posture-adjusted trigger percent and protects a wider recent-message window; Aggressive compacts earlier; Default stays unchanged on both levers. should_compact comparison logic is untouched. Also update the /harness panel copy now that the compaction strategy is consumed at runtime: the header no longer claims the posture is inert (was 'preview - does not change runtime behavior yet') and the compaction row is marked (live). Other knobs (max_subagents/tool_surface/safety) remain preview.
…n#2693 PR3b) When posture.max_subagents > 0, use it as the default concurrent sub-agent cap (still clamped to MAX_SUBAGENTS). Zero keeps the config path byte-identical. Explicit CLI --max-subagents wins via a dedicated override flag, and model/config changes sync the live engine through SetSubagentRuntimeConfig so App and engine stay aligned.
…own#2693 PR3c) ReadOnly intersects the Mode-built registry and MCP list with read-only tools only; Full/Auto stay no-ops. Execution denies non-read-only calls, and ensure_advanced_tooling skips code_execution/js_execution under ReadOnly so the model-visible catalog cannot re-widen after filtering.
New main-side test helpers lacked the PR3b field and failed to compile.
|
Thanks @SamhandsomeLee for taking the time to contribute. This repository is observing a maintainer-managed PR intake gate in dry-run mode, so this pull request is staying open. This note helps maintainers prepare the allowlist before any enforcement is considered. Please read |
Owner
|
wow cool! let me check this out and work to get it in this version. thank you for your hard work on this! |
Contributor
Author
Thank you! I'm glad it looks useful. This is my CodeWhale contribution, and I’m happy to help refine it or adjust anything needed so it can fit well into this version. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #2693 as one reviewable stack on latest
main: deterministic harness resolution, read-only status surfaces, then runtime wiring for compaction / sub-agent concurrency / tool surface.Slice map
resolve_harness(specificity + user-over-builtin + stable tiebreak; default Standard when unmatched). Oldresolve_harness_profilekept./harnessstatus + footer posture chip (preview only; drops first under width pressure).HarnessResolutionon the session and inject posture intoEngineConfig(carry path for later consumers).compaction_strategyinto compaction timing + protected window (Defaultidentity;PrefixCachelater/wider;Aggressiveearlier).max_subagentsinto runtime concurrency (posture when> 0; explicit CLI--max-subagentswins viamax_subagents_cli_override; live sync on model/config changes).tool_surfaceinto catalog/registry (never widens); ReadOnly filters + execution-time denial;ensure_advanced_toolingdoes not re-add exec tools under ReadOnly.Non-goals / hard boundaries
tool_surfaceonly narrows; Full/Auto are no-ops relative to the existing catalog.Commits (onto current
main)fix(tui): anchor compaction trigger to the input budget ceilingfeat(config): add deterministic harness profile resolverfeat(tui): add read-only harness status displayfeat(tui): carry harness resolution on sessionfeat(tui): wire harness compaction strategy into compaction (#2693 PR3a)feat(tui): wire harness max_subagents into runtime concurrency (#2693 PR3b)feat(tui): wire harness tool_surface into catalog (never widens) (#2693 PR3c)fix(tui): fill max_subagents_cli_override in TuiOptions after rebaseEvidence (PR3a)
Method. Same interactive workload twice on
deepseek-v4-flash(YOLO) withauto_compact_threshold_percent = 10. PrefixCache arm = built-in CacheHeavy seed fordeepseek-v4*; Default arm forced via user[[harness_profiles]](kind = standard,compaction_strategy = default). Metrics from/cache statsand/status.Result. Cache-heavy posture kept the pinned prefix more stable (2 vs 4 rewrites), raised session prompt-cache hit rate 90.8% → 94.6%, and cut cache-miss tokens ~32%. Even with more input processed (8.0M vs 7.0M), cost was lower (¥0.74 vs ¥0.90).
Deterministic lever check (base 80% on
deepseek-v4-pro): PrefixCache raises trigger 589,466 → 648,412 tokens and widens protected window 4 → 8; Aggressive lowers to 501,046; Default is byte-for-byte identity (589,466 / window 4).Caveats. One run per arm (directional, not exact). Compaction count proxied by prefix-change count on Windows (tracing logs not flushed;
compaction summary call:unavailable).timing-onlyvstiming+pathnot split; numbers above aretiming+path.Testing
cargo fmt(local; rebased branch)cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-featurescargo test --workspace --all-featurescargo test -p codewhale-tui --bin codewhale-tui --locked harness_→ 18 passed (after rebase onto latestmain)cargo test -p codewhale-config(resolver suite; please re-run in CI / before merge)Checklist
/harness, posture refresh, max_subagents CLI pin, tool_surface filter/deny)/harness, footer posture chip; PR3a interactive evidence above)