Repository navigation
fix: Keep Per-Conversation Context and the Summary at a Fixed System Position - #613
Shabonasar wants to merge 6 commits into
Conversation
…Position Adds conversation_instructions: context that is stable within a conversation. It stays in the system message right after the static instructions and is never moved: its own content block with a cache breakpoint for Anthropic and OpenRouter, a block followed by a cache point for Bedrock, appended text for other providers. additional_instructions keeps its per-turn behaviour. A host that passes conversation_instructions gets the compaction summary, both when created inside a run and when restored as initialSummary, as one stable user message right after the system message, with a breakpoint of its own while the request has one left. Token accounting counts it as instruction overhead. Both paths render the same carrier text, so the requests after a compaction form one continuous prefix. Hosts that do not pass the field keep the earlier placement and accounting. Adds a prompt cache harness (src/specs/prompt-cache) that runs declarative scenarios through Run and each provider class, captures the serialized request before the network, and asserts the cache invariants per provider.
With prompt caching on Bedrock, additional_instructions was appended to the system message after its last cachePoint. Bedrock reads tools, system and messages in that order, so a tail that changes between turns sits in front of the whole history and every message checkpoint misses. The tail now moves to a message right before the newest user message, as for Anthropic and OpenRouter, with one cachePoint at the end of the history in front of it and one at the end of the request. The conversation block keeps its own cachePoint only while the four-checkpoint limit leaves room for it (tools, stable block, history, request end come first). Without prompt caching the tail stays in the system message.
…Marks Two Messages The graph can place a second message checkpoint for the latest human message. Beside the system checkpoints and the tool list that overran the limit of four in two places: - the conversation block kept its own cachePoint only on the moved-tail path; the Bedrock system message now drops it whenever the checkpoints already in the messages leave no room, with or without a moved tail; - the opening message of the history in front of a moved tail kept the graph's checkpoint on top of the two the tail path places; it is stripped now.
|
Added: Bedrock tail placement (reported for Bedrock in LibreChat-AI/LibreChat#16877). With Compatibility with #631: the system block budgets its checkpoints against every checkpoint already placed in the messages, so #613 + #631 stays within 4 with or without a moved tail. With a moved tail #613 places its own two message checkpoints and never marks the tail; #631's second checkpoint applies to requests without one. Note: Delta: Shabonasar/agents@7676461...83eea26 — 872 tests pass on this branch, 1935 with #631 merged on top (4 failed the at-most-4 check before the fix). Not run against a live Bedrock endpoint. |
…he cached prefix Per-conversation instructions keep a fixed system position, the compaction summary has one placement, and the Bedrock dynamic tail moves in front of the newest user message within four cachePoints. Tool results keep the same bytes when the tail marker moves.
|
Reduced to simplify dependencies and the review: based on
|
|
Branch updated with a merge commit (no file changes) so that the agents PRs of LibreChat-AI/LibreChat#17102 merge together with a plain |
Summary
With Anthropic or OpenRouter prompt caching,
AgentContextmoves the dynamic system part into a user message before the newest human message. Context that is fixed for a whole conversation (start time, memory, file lists) moves with it on every turn, so the previous exchange is written to the cache again. A summary created by a compaction inside a run is sent as a user message, while the next run renders the same summary insystem, so every compaction costs a second prefix rebuild.On Bedrock with
promptCachethe dynamic part stays at the end of the system text, in front of the whole history, so a per-turn change there misses every message cache point (reported in LibreChat-AI/LibreChat#16877).This PR adds
AgentInputs.conversation_instructions, kept in the system message right afterinstructionsand never moved, and gives the summary one fixed placement when the field is set. On Bedrock withpromptCachethe dynamic tail moves behind the cached history, as it already does for Anthropic, within four cache points. A tool result keeps the same bytes when the tail cache marker moves on. As a result, request N+1 starts with request N up to the per-turn part, and after a compaction the run and the next run send the same prefix.Related to LibreChat-AI/LibreChat#16877 (Bedrock part) and LibreChat-AI/LibreChat#17102 (tracking; design discussion LibreChat-AI/LibreChat#17481). Base:
main, no dependency on other PRs.How it works
Sstatic instructions,Tper-conversation context,Dper-turn context,bpcache breakpoint,cpBedrock cache point.promptCachepromptCacheinstructionsconversation_instructions(new)additional_instructionsinitialSummary) is one user message right after the system message, stable until the next compaction, and counts as instruction overhead.setSummary'sprecedesMessagesis ignored then.tool_resultthat is one plain text block is sent as its string; OpenRouter sends every non-empty tool message as one text block. Both match what the tail cache marker produces, so the bytes stay the same before and after the marker moves.Type of change
Testing
Tested environments/configuration:
/api/agents/chat) with the LibreChat series from Tracking: stable prompt prefix cache across turns, conversations and providers LibreChat#17102 and all five SDK PRs (fix: Keep Per-Conversation Context and the Summary at a Fixed System Position #613, feat: Add a Fixed Tool Dispatcher for Mid-Conversation Tools #614, fix: Restore Tool Rounds Exactly as the Live Request Sent Them #615, feat: Replay Signed Reasoning in Restored History #619, fix(tools): persist tool output reference labels and keep keys unique across runs #629), cache TTL 1h.cache_controlmarkers, no API errors.Automated tests:
anthropic,google,vertexaillm.spec.ts).AgentContext.test.ts: Bedrock tail moved out ofsystem, history marked at both ends of the moved tail, the conversation block's cache point only while one of four is left, a compaction summary and an initial summary rendered as the same first message.tool-id-normalization.test.ts,messages.test.ts: tool result bytes stay the same when the marker moves.durability-checkpoint.integration.test.tsandpersistent-handoffs.test.ts, which need MongoDB and fail onmaintoo.packages/apisrc/agents6913 passed (1 failure inhandlers.shutdown.spec.ts, a shutdown timing test);apiserver/controllers/agents1371 passed.Risk / compatibility
conversation_instructionsis opt-in. Hosts that do not pass it keep the earlier layout and summary placement.promptCachenow sendsadditional_instructionsas a message before the newest user message instead of at the end ofsystem(the fix for [Bug]: web_search runtime context stamps the system prompt, defeating prompt caching (regression) LibreChat#16877). The tool result shape changes for Anthropic and OpenRouter. In both cases, existing caches are written once more after the upgrade.Checklist