feat(browser): capture ChatGPT's own conversation record as evidence - #399
Open
frontierkodiak wants to merge 4 commits into
Open
feat(browser): capture ChatGPT's own conversation record as evidence#399frontierkodiak wants to merge 4 commits into
frontierkodiak wants to merge 4 commits into
Conversation
Oracle's answer capture is a rendering of what ChatGPT displayed. For most
answers that is the same thing as the answer; for notation it is not. On a live
Pro run in a ChatGPT project, the captured Markdown for a math-heavy turn
differed from the provider's own record of that same turn: `\,` lost its
backslash and `\mathcal{F}_s` came back as `\mathcal{F}*s`. Nothing failed, no
fallback fired, and every keyword-level check passed — the text simply was not
what the model wrote.
`--browser-capture-provider-native` (off by default) additionally fetches
`/backend-api/conversation/<id>` from inside the authenticated page and saves two
files beside the run's other artifacts:
- the conversation document, verbatim — the bytes are written from the same
string that was hashed, with no parse-and-reserialize in between;
- an evidence file from a SECOND, independent fetch, normalized and hashed in the
page so only digests cross the boundary. Node never sees the second body, so a
mistake on this side cannot make the two agree by construction.
The run's own answer is then compared to those digests, and the result recorded
as matched / divergent / unknown. That replaces a length heuristic with the
provider's bytes: "this transcript is the provider's text" becomes checkable
rather than assumed.
Deliberate limits:
- Capture never gates an answer. `/backend-api/*` sits behind bot mitigation that
can return 403 to an in-page fetch while the user is logged in, so every
failure is a typed reason and a normal result. A conversation with no id —
temporary chats, or a run whose URL never settled — is `unavailable`, not an
error.
- Document-level hashes of the two fetches are recorded, never gated: the backend
document carries volatile nested metadata and can differ between fetches at
identical turn content. The per-turn comparison is the load-bearing one.
- The bearer token from /api/auth/session is used in the page and never returned
or logged, per the existing note in navigation.ts.
- Payload is drained in bounded chunks with a ceiling and a timeout, and
`exceptionDetails` is checked, so an in-page throw is reported rather than
collapsed into an empty result.
The in-page normalization is checked byte-for-byte against the reference
implementation it must agree with, over a fixture whose expected digests that
reference produced — including the JSON-fallback branches where Python's
sort_keys/ensure_ascii dumps and its int-vs-float rendering diverge from
JSON.stringify.
Claude-Session: https://claude.ai/code/session_01HsXirqcfqtr1Cae9zYCLDk
… not the second fetch The evidence file reported the second fetch's document hash as though it described the raw JSON sitting next to it. Verifiers use that field to confirm the file on disk was not altered between capture and ingest, so pointing it at a different fetch made the check fail for a reason unrelated to what it tests — and fail intermittently, since the two fetches sometimes agree and sometimes do not. Both Quiet conversations were captured twice while working on this: the first pass produced matching document hashes for one and differing hashes for the other; the second pass produced differing hashes for both, over identical turn content. That is the nested-metadata volatility this format already expects, and it is exactly why document-level equality is a poor fidelity criterion. `raw_backend_api_json` now describes the document actually on disk. The second fetch moves to its own `independent_fetch` block, where its document hash is a volatility record rather than a criterion, and its per-turn digests remain what the fidelity comparison is built on. Claude-Session: https://claude.ai/code/session_01HsXirqcfqtr1Cae9zYCLDk
The flag's plumbing landed with the capture feature but its option registration did not, so the config field existed and nothing could set it from the command line. Belongs squashed into the capture commit before this goes upstream. Claude-Session: https://claude.ai/code/session_01HsXirqcfqtr1Cae9zYCLDk
Contributor
|
🦞👀 Pull request received. I will update this pull request when review starts. |
clawsweeper
Bot
added
merge-risk: 🚨 security-boundary
Contributor
|
Codex review: needs real behavior proof before merge. Reviewed August 18, 2026, 3:04 PM ET / 19:04 UTC. ClawSweeper reviewWhat this changesThe PR adds an opt-in browser flag that fetches ChatGPT’s conversation document from the authenticated page, stores it as an artifact, and compares independently computed per-turn digests against Oracle’s captured answer. Merge readiness⛔ Blocked until real behavior proof is added - 12 items remain Keep open: the feature has a direct remote data-export boundary and an incorrect evidence metadata flag that should be resolved before merge. Priority: P1 Review scores
Verification
How this fits togetherOracle browser mode drives an authenticated ChatGPT session and stores answers and session artifacts. This optional capture runs after an answer, fetches the provider’s conversation record, and emits raw/evidence artifacts plus a fidelity summary. flowchart LR
A[ChatGPT browser session] --> B[Captured answer]
B --> C[Optional native capture]
C --> D[Authenticated conversation fetch]
D --> E[Raw conversation artifact]
D --> F[Independent digest evidence]
E --> G[Bridge artifact transfer]
F --> H[Fidelity summary]
Decision needed
Why: The branch turns a browser configuration field into an export of authenticated conversation history through the bridge; this credential scope cannot be chosen safely by patch mechanics alone. Before merge
Findings
Agent review detailsSecurityNeeds attention: The new authenticated capture can expose full provider conversation data through the remote bridge unless its activation is host-authorized. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Keep local CLI capture opt-in, make remote export require an explicit host-controlled authorization policy, correct the materialization flag, and add a focused remote-boundary regression plus redacted browser proof. Do we have a high-confidence way to reproduce the issue? Yes — source establishes the path from remote browser configuration through capture to bridge-registered raw artifacts; this does not require a live credential to verify the boundary. Is this the best way to solve the issue? No — enabling the field through the existing bridge configuration path silently broadens credential scope; a host-controlled export permission is the safer design. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 083bba7e61f4. LabelsLabel changes:
Label justifications:
EvidenceSecurity concerns:
What I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Oracle's answer capture is a rendering of what ChatGPT displayed. For most answers that is the same thing as the answer; for notation it is not.
On a live Pro run in a ChatGPT project, the captured Markdown for a math-heavy turn differed from the provider's own record of that same turn:
\tfrac{1}{2}\,\Gamma→\tfrac{1}{2},\Gamma(escape dropped)\mathcal{F}_s→\mathcal{F}*s,\sum_{n=0}→\sum*{n=0}\(…\)→ bare parensNo fallback fired — this was the preferred copy-button path — and every keyword-level check passed. The text simply was not what the model wrote. A sentinel built from "are the markers present" structurally cannot see a transformation that preserves markers, and notation is where those hide.
Approach
--browser-capture-provider-native(off by default) additionally fetches/backend-api/conversation/<id>from inside the authenticated page and saves two files beside the run's other artifacts:The run's own answer is then compared against those digests and recorded as
matched/divergent/unknown. That replaces a length heuristic with the provider's bytes.It discriminates rather than rubber-stamps: on the same conversation, a math-heavy turn reports
divergentwhile a prose follow-up reportsmatched.Deliberate limits
/backend-api/*sits behind bot mitigation that can return 403 to an in-page fetch while the user is logged in, so every failure is a typed reason and a normal result. A conversation with no id — a temporary chat, or a run whose URL never settled — isunavailable, not an error.trueandfalse. The per-turn comparison is the load-bearing one./api/auth/sessionis used in the page and never returned or logged, per the existing note innavigation.ts.exceptionDetailsis checked, so an in-page throw is reported rather than collapsed into an empty result.Tests
The in-page normalization is checked byte-for-byte against an independent reference implementation, over a fixture whose expected digests that reference produced — covering text, code, thoughts, reasoning_recap, multimodal_text, and the unknown-content-type JSON fallback, including where Python's
sort_keys/ensure_asciidumping and its int-vs-float rendering diverge fromJSON.stringify. Plus the failure paths: no conversation id, a bot-mitigation challenge, and an in-page exception.Full suite green: 1766 passed / 43 skipped;
docs checkok.Note
Enabling this over the bridge additionally needs
captureProviderNativeon the accepted client-config list; that is one line, and it lands with #398 rather than here to keep the two reviewable apart.