Skip to content

perf(context_engine): replay a turn's skill menu and scratch byte for byte - #1334

Draft
LivXue wants to merge 3 commits into
mainfrom
perf/replay_turn_scratch_and_scent
Draft

LivXue wants to merge 3 commits into
mainfrom
perf/replay_turn_scratch_and_scent

Conversation

@LivXue

@LivXue LivXue commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

A provider's cache is a prefix cache: the next turn reuses what the previous one sent only when the bytes match. Two pieces of the turn's user message carried per-turn text that the session write dropped, so the next turn's request differed from the previous one and the cache broke at every turn boundary.

The skill menu rides in the same paragraph as the runtime-context lines, but the assembler recorded the lines alone. The turn scratch (the recalled memory and routed skills for the turn) rides at the end of the user message and was cut and discarded. Both are now filed with the message and rejoined on the way out: the block as runtime_context, the scratch as turn_scratch, the menu inside the block. What the provider saw is what the next turn sends.

The stored content keeps the user's words alone. The transcript, the novelty window and every other reader of the session log see the question, not the recall.

Every join is located by what identifies it, never by position. A notice put in front of the message after assembly moves the block off the content's start, and the replay found it by position before, which joined the block twice. The scratch is filed with the offset it stood at, because a hook's note lands on the message after the scratch was appended, so the scratch is not always last.

The scratch's tag said it was "not conversation history". It is now filed and replayed, so the tag says what it is: the recall of the turn it was asked in. The ppt engine's ledger keeps its own copy of the two spellings, pinned against the trunk by a test, and it moves with them.

A review round found one gap in the first form of this: a user entry filed with no runtime_context (recovery injection leaves the block in the content, so the session write files none) returned early from the replay and lost its filed scratch. The scratch is now rejoined for every user entry; the early return covers only the block.

Follows #1294, which filed the runtime-context lines and left these two behind.

Type

  • Fix
  • Feature
  • Docs
  • CI / tooling
  • Refactor
  • Other

Verification

  • Relevant tests pass locally
  • Relevant lint / type checks pass locally
  • User-facing docs or screenshots are updated when needed

Commands, run with the branch's own venv (Python 3.12.13, all extras):

  • The new tests, over the real assembler, the real session write and the real history projection: tests/test_context_replay_bytes.py and tests/test_context_replay_join.py. The first one drives two turns and compares the bytes of the first turn's question as it was sent against the copy the second turn carries.
  • tests/test_context_turn_scratch.py, tests/test_context_runtime_context.py, tests/test_token_estimation.py, tests/test_ppt_engine_ledger.py, tests/test_context_compacting_runtime.py, tests/test_agent_loop_reply_room.py, tests/test_curator_context_engine.py, tests/test_runtime_checkpoint.py: 381 passed on the rebased head.
  • The review fix has two tests of its own, both red before it: the unit shape the recovery path files, and an end-to-end run through the real TurnPathMixin._inject_recovery_block, the real _save_turn and the real projection.
  • 10 mutants of the new logic (the scratch not rejoined, the block joined twice, the scratch joined at the wrong end, the filed scratch ignored, the menu not recorded, the menu not sent, the scratch not filed, the block filed without the menu, the scratch dropped when no block is filed, the scratch joined after the block): all killed.
  • scripts/coverage_gate.py diff --base-ref origin/main over a targeted run: 90.67% of changed lines (68/75), over the 90% threshold.
  • A local probe that drives the real assembler, the real session write and the real history projection over fourteen turns, measuring the common prefix between consecutive requests: the prefix now grows by the whole request per turn (1,749 bytes at the probe's size), against 328 bytes before, which was the part that never changed. The request grows by the same replayed text; the stored session grows by the filed scratch.
  • Full suite pytest -q (repo defaults): 40580 passed, 267 skipped, 13 failed. Twelve are the environment failures that also fail on main, measured there the same day: the five proxy tests in test_config_update_providers.py (the box exports an HTTP proxy), the four in test_evolver_git_ops.py, test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing (run as root), test_knowledge_parser_excel.py::test_a_legacy_workbook_is_converted_and_read, and test_install_script.py::test_resolve_node_dir_answers_each_case_it_exists_for. The thirteenth, test_cli_agent_commands.py::test_an_empty_message_file_is_a_usage_error, passes alone (47 passed) and under four workers twice; it does not touch the code this change edits. 0 introduced.
  • ruff check and ruff format --check on the changed files, ty check on the changed modules, lint-imports: clean.

Risk

  • Security impact considered

  • Backward compatibility considered

  • Rollback path is clear for risky changes

  • The scratch's tag text changes, so every live session sends different bytes and every open cache misses once. No request changes shape.

  • The session log grows by the filed scratch, roughly the size of one recall per turn. The recall was previously dropped; it is now kept, which is what makes the replay exact.

  • The model re-reads the recalls of the earlier turns, each with its own recall text, since each turn's message replays as it was sent.

  • A message filed before this change has no turn_scratch field and replays exactly as it did.

  • Rollback: revert the commits; no data or config migration.

Related Issues

Follows #1294.

@LivXue
LivXue requested review from 0xKT and gloryfromca November 17, 2026 09:16

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the recovery-notice path drops a filed turn scratch from replay.

I independently reviewed the full change and found one inline blocker. The ordinary string recovery path contradicts the byte-for-byte replay contract: the session writer files the scratch, but the history projection does not restore it.

Coverage: I checked AGENTS.md and the runtime terms in CONTEXT-MAP.md/CONTEXT.md; read the complete diff; traced assembly, recovery injection, persistence, both history projections, token estimation, and the relevant provider cache shaping; inspected the base and commit history; checked backward-compatibility behavior and the changed tests for weakened assertions; and checked the runtime layer boundaries touched here.

Verification: uv run pytest -n0 tests/test_context_replay_bytes.py tests/test_context_replay_join.py tests/test_context_runtime_context.py tests/test_context_turn_scratch.py tests/test_token_estimation.py -q passed: 172 tests in 9.29s. A direct reproduction through AgentLoop._save_turn produced a stored turn_scratch without runtime_context, then replayed_content omitted the scratch (MATCH: False).

Comment thread raven/context_engine/runtime_context.py Resolved
@LivXue LivXue changed the title perf(context_engine): file the skill menu and the turn scratch so a turn replays byte for byte perf(context_engine): replay a turn's skill menu and scratch byte for byte Nov 17, 2026
LivXue and others added 3 commits November 17, 2026 09:46
…urn replays byte for byte

A provider's cache is a prefix cache: the next turn reuses what the
previous one sent only when the bytes match. Two pieces of the turn's
user message carried per-turn text that the session write dropped, so
the next turn's request differed from the previous one and the cache
broke at the turn boundary.

The skill menu rides in the same paragraph as the runtime-context
lines, but the assembler recorded the lines alone. The turn scratch
rides at the end of the message and was cut and discarded. Both are now
filed with the message and rejoined on the way out: the block as
`runtime_context`, the scratch as `turn_scratch`, the menu inside the
block. What the provider saw is what the next turn sends.

The stored `content` keeps the user's words alone. The transcript, the
novelty window and every other reader of the session log see the
question, not the recall.

Every join is located by what identifies it, never by position. A notice
put in front of the message after assembly moves the block off the
content's start, and the replay found it by position before, which
joined the block twice. The scratch is filed with the offset it stood
at, because a hook's note lands on the message after the scratch was
appended, so the scratch is not always last.

The scratch's tag said it was "not conversation history". It is now
filed and replayed, so the tag says what it is: the recall of the turn
it was asked in. The ppt engine's ledger keeps its own copy of the two
spellings, pinned against the trunk by a test, and it moves with them.

Measured on a local probe that drives the real assembler, the real
session write and the real history projection over fourteen turns: the
common prefix between consecutive requests grows by the whole request
per turn (1,749 bytes at the probe's size), against 328 bytes before,
which was the part that never changed. The request itself grows by
the same replayed text, and the stored session by the filed scratch.

The tag change alters the bytes every live session sends, so every
open cache misses once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ruff format wraps one long assertion in the new test file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
replayed_content returned early whenever a user entry carried no
runtime_context field, so the scratch filed on that entry was never put
back. The recovery-notice path reaches exactly that shape:
_inject_recovery_block prepends its notice to the assembled message, so
the session write leaves the block in content and files none, while it
still cuts and files the scratch. The replayed request then lost the
recall the provider had seen, which is neither byte-identical nor the
same question.

The early return now covers only the block. The scratch is rejoined for
every user entry, before the block, because where it stood is an offset
into the message's own words.

Reported by gloryfromca on #1334, reproduced through the real session
writer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LivXue
LivXue force-pushed the perf/replay_turn_scratch_and_scent branch from 9b21225 to 6348c37 Compare November 17, 2026 09:47
@LivXue
LivXue requested a review from gloryfromca November 17, 2026 09:47

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

The new revision fixes the outstanding recovery-path defect. I verified the delta against the full current PR diff and the surrounding assembly, recovery injection, persistence, history projection, and token-estimation callers. I also rechecked the applicable AGENTS.md and CONTEXT.md contracts, backward compatibility for entries without filed fields, architecture boundaries, and that the added tests strengthen rather than replace prior coverage. I found no new issue worth raising.

Verification: the original direct reproduction now reports MATCH: True; uv run pytest -n0 tests/test_context_replay_bytes.py tests/test_context_replay_join.py tests/test_context_runtime_context.py tests/test_context_turn_scratch.py tests/test_token_estimation.py -q passed, 174 tests in 9.21s. The original review thread is resolved.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: filing every turn's scratch and replaying it on every later request makes the history grow by the whole recall each turn; on a plain chat session the request is 2.5x base by turn 50, compaction comes much sooner, and recall the memory store has since dropped keeps reaching the model.

_save_turn now files the scratch on each user entry (raven/agent/loop/turn_path.py:4376-4382) and replayed_content puts it back on every later request (raven/context_engine/runtime_context.py:105 -> scratch.py replayed_scratch). Nothing bounds it except compaction; the pruner never touches user messages. Base cut the scratch and discarded it, and its own docstring named this outcome as the reason ("the recalled text would accumulate in history forever").

Measured at 6348c37 merged onto 27becfa against base 27becfa, with the real ContextAssembler, the real AgentLoop._save_turn and SessionManager, a stub backend recalling 5 hits of ~500 chars plus the selected user.md sections (about 700 tokens of scratch a turn), one session, 50 turns, every third turn with a tool read:

request tokens      t1     t5     t20     t50     replayed recall at t50   session file
base               3298   4807   11276   24502    0                        122,458 B
head               3294   7913   26034   62588    34,936 (56%)             280,579 B
previous request fully reused at the next turn boundary: base 0/49, head 49/49

The cache claim is real: head re-sends the previous request byte for byte. But what it buys is the menu and one scratch per boundary, while every later request now carries every earlier scratch. Cached tokens are cheaper, not free, and the extra 38k tokens a request also fill the window: at a fixed window the filed recall alone reaches the compaction trigger many turns sooner and keeps re-reaching it, each compaction dropping the cached prefix and paying a summary call whose input is now mostly replayed recall. For a change titled perf, the default chat path gets more expensive, not less; the gain shows only on tool-heavy turns.

The same filing changes what the model is told: a fact recalled on turn 1 and then dropped from the store (a "forget this", a profile refresh) is still sent on every later request until compaction, inside the recall fence, and stays in the session file. Same probe: turn 1 recalls F, the backend and user.md then drop it; head's turn 2 request carries F in the history user entry, base's does not.

Shapes that keep the cache gain without the growth, for the author to choose from: file only the skill menu (small, and the part that sits in the runtime paragraph) and keep discarding the scratch; or replay the scratch for the most recent user turn only (the boundary that breaks the cache) and drop older ones; or cap the filed scratch. Whichever way it goes, CONTEXT.md's Turn Scratch entry (CONTEXT.md:1706, "never session.messages", and :1711-1713) has to match: today it says the opposite of the code, and of the Runtime Context entry this PR edited.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: resolve the unbounded historical scratch replay before merge.

This evidence invalidates my earlier clean stance. I independently confirmed the core mechanism: a 50-user-message stored history projects all 50 filed scratches back into the request, with no age/count cap before compaction; in a synthetic 2,800-character-per-turn check, the stored-message estimate grew from 200 tokens without scratches to 70,490 with them. The existing Turn Scratch contract also still says the scratch never reaches session.messages, which confirms this storage and semantic expansion was not reconciled across the design.

The replay/runtime/token suite still passes (174 passed in 9.34s), but those tests assert byte-identical replay and therefore do not exercise whether retaining every prior recall is affordable or semantically safe. I am withdrawing my clean signoff while this thread remains outstanding.

@0xKT

0xKT commented Nov 17, 2026

Copy link
Copy Markdown
Member

Not a blocker: three more effects of filing the scratch, for whichever shape the blocking thread lands on (6348c37 merged onto 27becfa; base 27becfa is the control).

Severity of this one finding, not a verdict on the pull request. Any blocking findings from this pass are review threads on the changed files; GitHub renders those collapsed, as a file name with no text.

  1. Bug-report bundles now ship every turn's recall. collect_bundle copies the session file verbatim (raven/trajectory/bundle.py:319-326, unchanged), and the file now carries turn_scratch on every user entry, so a bundle collected for one attempt carries the recalled memory of every turn in the session, later turns included. redact.py scrubs credentials only, so personal memory text passes. The commit body says every reader of the session log sees "the question, not the recall"; the verbatim copiers are the exception.

  2. The memory backend's store() now receives the recall. The after-turn harness peer gets the turn through _without_scratch (turn_path.py:327-337, "this turn's input, not its record"), but the backend-store peer gets session.messages[_turn_start():] (turn_path.py:3727 -> organ_glue.py:343-351), which now holds turn_scratch, the text recalled from that same backend. The bundled EverOS backends drop unknown keys, so nothing breaks in-tree, but a backend that stores rows whole would feed its own recall back into memory. The guard in tests/test_context_turn_scratch.py went vacuous: it now checks user content only on lines starting with {, and json.dumps(backend.stored) is one line starting with [[.

  3. Docs say two different things. Besides the Turn Scratch entry named in the blocking thread, the new sentence "Every join is located by what identifies it, never by position" (CONTEXT.md:1679) and replayed_scratch's docstring do not hold: it places the scratch at the offset or index recorded at save time. runtime_context.locate, added for locating by identity, has no caller outside tests.

@LivXue
LivXue marked this pull request as draft November 17, 2026 12:02
@LivXue

LivXue commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

Windows compatibility review — 6348c37631

Reviewed at head 6348c37631d2abc03c2dbbe5319791ab42810d08 (base 27becfa2fa; draft PR). Scope: the PR's own diff (raven/context_engine/scratch.py, raven/context_engine/runtime_context.py, raven/context_engine/assembler.py, raven/agent/loop/turn_path.py, raven/utils/tokens.py, plugins-dist/ppt-engine/raven_ppt/plugin/ledger.py, five test files), with the new split_turn_scratch / replayed_scratch offset logic read against every reader of the stored content.

No Windows compatibility issue found.

The change files the turn scratch beside the user's words (turn_scratch + a character offset) instead of dropping it, and the replay puts it back at that offset; the runtime-context block, menu included, is now the recorded _runtime_context under the same carries_runtime_context check. All of it is str/list/int work: the offset is a Python character index into a string stored in the same JSON record, so it round-trips identically on Windows (JSON escapes \n as two characters, the session is read with universal newlines, and nothing compares bytes against a file's on-disk encoding). locate and _block_holding find a recorded text by identity rather than by position, which is the right shape for the recovery-notice case they cite. The plugin's copy of the tag string is updated to the same sentence; the trunk spelling is pinned by the tests, and the plugin reads it only as a literal.

Verification (native Windows host, Python 3.12.13, shared venv, detached worktree pinned to the head):

  • python -S runner.py tests/test_context_replay_bytes.py tests/test_context_replay_join.py tests/test_context_runtime_context.py tests/test_context_turn_scratch.py tests/test_token_estimation.py -q → 174 passed in 29.26s
  • python -S runner.py tests/test_context_stable_prefix.py -q → 18 passed in 25.22s

Coverage gaps: no live provider call, so the cached-prefix gain the byte-for-byte replay exists for is reasoned from the assembled payload rather than measured against a real provider cache; the session-file round trip is exercised through the replay tests, not by writing and reloading a gateway session on this host.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants