feat(browser): persist thinking-effort selection evidence - #397
Open
frontierkodiak wants to merge 3 commits into
Open
feat(browser): persist thinking-effort selection evidence#397frontierkodiak wants to merge 3 commits into
frontierkodiak wants to merge 3 commits into
Conversation
`ensureThinkingTime` verified the requested effort tier and then threw the result away, so a completed run left no machine-checkable proof of the tier it submitted at — only an unstructured `[browser] Thinking time: …` log line. That gap is load-bearing rather than cosmetic. `browser.modelSelection` cannot stand in for it: for a Pro-capable target the picker deliberately reports the requested model string as the resolved label, so `resolvedLabel === requestedModel; verified: true` is byte-identical whether or not the Pro effort row was ever selected. Any consumer asking "did this run answer at Pro effort?" had nothing to read. `ensureThinkingTime` now returns a `BrowserThinkingSelectionEvidence` record alongside its existing logging, and it is persisted as `browser.thinkingSelection` parallel to `browser.modelSelection`, crossing the remote bridge on `BrowserRunResult` and rendering in `oracle status`. `verified` is true only for the two statuses that positively observed the option's selected state. Strict (fail-closed) requests still throw before submit on every other outcome, so they never produce an unverified record — which makes a persisted strict record proof by refusal that no degraded tier was used. Claude-Session: https://claude.ai/code/session_01HsXirqcfqtr1Cae9zYCLDk
…ridged results `sanitizeResult` whitelists what crosses the bridge, which is the right shape — a result must never carry this host's pids, ports, or profile paths to a client on another machine. But the whitelist had drifted narrower than that rule: it also dropped `modelSelection`, `thinkingSelection`, `archive`, `tabUrl`, `conversationId`, and `promptSubmitted`. The effect is that a remote caller could not tell which model or effort answered their prompt — the evidence the picker verified was computed, logged on the host, and then discarded at the boundary — and could not bind the answer to a durable ChatGPT URL. A bridged answer arrived unattributable, and silently so: the fields are optional, so nothing failed. None of the added fields describes the host. The three that do stay explicitly undefined, and the test now asserts both halves of the rule so a future field cannot quietly pick the wrong side. Claude-Session: https://claude.ai/code/session_01HsXirqcfqtr1Cae9zYCLDk
Contributor
|
🦞👀 Pull request received. I will update this pull request when review starts. |
clawsweeper
Bot
added
P3
Contributor
|
Codex review: needs maintainer review before merge. Reviewed August 30, 2026, 9:33 PM ET / August 31, 2026, 01:33 UTC. ClawSweeper reviewWhat this changesThe PR persists and displays the selected ChatGPT thinking-effort tier for browser runs and returns associated conversation metadata through the remote result bridge. Merge readinessThe effort-evidence change remains absent from current main and is a coherent extension of existing browser model evidence. This branch is merge-conflicted with newer remote security work, so it needs a careful rebase before merge. Priority: P3 Review scores
Verification
How this fits togetherOracle’s browser runner selects a ChatGPT model and thinking tier before submitting a prompt, then stores run metadata for flowchart LR
A[Browser run configuration] --> B[ChatGPT effort picker]
B --> C[Selection evidence]
C --> D[Browser run result]
D --> E[Session metadata]
E --> F[Oracle status]
D --> G[Remote bridge client]
Before merge
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Port the focused evidence record and its browser, status, and bridge coverage onto current main while retaining the already-merged remote configuration boundary. Do we have a high-confidence way to reproduce the issue? Not applicable as a conventional bug reproduction: supplied terminal output demonstrates the new evidence record, and source inspection confirms current main still lacks it. Is this the best way to solve the issue? Yes, after rebase: a separate typed effort record is more reliable than inferring the selected tier from model-selection metadata or logs. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against bbc1b3b0261d. LabelsLabel changes:
Label justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
History |
clawsweeper
Bot
added
merge-risk: 🚨 compatibility
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ensureThinkingTimeverified the requested effort tier and then threw the result away, so a completed run left no machine-checkable proof of the tier it submitted at — only an unstructured[browser] Thinking time: …log line.That gap is load-bearing rather than cosmetic.
browser.modelSelectioncannot stand in for it: for a Pro-capable target the picker deliberately reports the requested model string as the resolved label, sois byte-identical whether or not the Pro effort row was ever selected. Anything asking "did this run answer at Pro effort?" had nothing to read.
Second,
sanitizeResultwas dropping the answer to that question at the bridge. Its whitelist correctly keeps host detail (pids, ports, profile paths) from crossing to a client on another machine, but it had drifted narrower than that rule and also droppedmodelSelection,thinkingSelection,archive,tabUrl,conversationId, andpromptSubmitted. A remote caller could not tell which model or effort answered their prompt, nor bind the answer to a durable ChatGPT URL — and silently, since the fields are optional.Approach
ensureThinkingTimereturns aBrowserThinkingSelectionEvidencerecord alongside its existing logging; it is persisted asbrowser.thinkingSelectionparallel tobrowser.modelSelection, crosses the bridge onBrowserRunResult, and renders inoracle status.verifiedis true only for the two statuses that positively observed the option's selected state (aria-checked/selected/data-state, or a composer pill matching the target tier). Strict fail-closed requests still throw before submit on every other outcome, so they never produce an unverified record — which makes a persisted strict record proof by refusal that no degraded tier was used.The added
sanitizeResultfields describe the conversation, not the host; the three that describe the host stay explicitlyundefined, and the test now asserts both halves of the rule so a future field cannot quietly pick the wrong side.Real behavior
A live Sol run at Pro effort now records:
The second line is the one that was previously unavailable. On a
--followup, model selection is legitimately reportedskipped/verified=no(a resumed conversation keeps its model) while effort evidence still readsverified=yes— the record stays honest about which of the two it re-proved.Tests
Four added covering the evidence contract, including that a strict Pro request never resolves to an unverified record across every degraded picker outcome. Two existing assertions that pinned the old
voidreturn were updated to assert the evidence shape instead. Bridge test asserts evidence and identity cross while host detail does not.Full suite green: 1766 passed / 43 skipped.