Repository navigation
Bedrock prompt cache misses on the next turn: reasoning is not replayed and dynamic instructions sit after the system cache point #617
Description
Summary
With Bedrock Claude models, promptCache: true and extended thinking, the first call of a turn sent shortly after the previous one (well inside the 5 min TTL) usually reads only the system prompt from the cache and writes the whole conversation again. Two independent causes:
- C. Reasoning is not replayed, so the rebuilt history differs from what was cached. During a run, assistant steps carry signed
reasoningContent. On the next turn the history is rebuilt from the stored message:formatAgentMessagesmovesthinkparts toadditional_kwargs.reasoning_content, andconvertToConverseMessagesdoes not emit them (with or without a signature). From the first assistant step of the previous turn onward the bytes differ, so the previous turn's tail checkpoint can never match. - A. Dynamic instructions follow the system cache point on Bedrock.
AgentContext.buildSystemMessageplaces the dynamic block (for example the code-execution file note, the web-search date context, memories) after the systemcachePoint. For Anthropic and OpenRouter it is moved next to the last human message (shouldMoveDynamicInstructions), but not for Bedrock. Any change in that block, such as a newly attached file or the hour rolling over, invalidates every message after the system prompt even when the history is identical.
A third factor makes (1) worse: Bedrock gets a single message checkpoint at the tail (addBedrockTailCacheControl), and Bedrock only looks back about 20 content blocks from each cachePoint. The one prefix that does still match (up to the human message that opened the previous turn, written as the tail of that turn's first call) is out of reach once the previous turn had a few tool steps.
Reproduction
- Bedrock Claude model with
promptCache: trueand reasoning enabled, an agent with at least one tool. - Turn 1: a request that makes the model call a tool 5 or more times.
- Within a minute, turn 2: any short follow-up.
- Usage of the first call of turn 2:
cacheReadInputTokensequals the system prompt size, andcacheWriteInputTokenscovers the entire history. - Variant for (A): same, but attach a file on turn 2 so the code-execution file note changes. Even with a one-call turn 1, turn 2 reads only the system prompt.
A unit-level repro without network: build turn 1 in memory (AIMessages with signed reasoning_content plus tool calls), run each call through addBedrockTailCacheControl + systemRunnable + convertToConverseMessages, then rebuild turn 2 from the persisted payload via formatAgentMessages, and compare the flattened Converse blocks against a prefix cache with a 20-block lookback.
Suggested fixes
- A: on Bedrock with prompt caching, move the dynamic block out of the system message into a message right after the latest human message (as is already done for Anthropic), so it sits after the message checkpoints. It is worth wrapping it in an explicit marker, because Converse merges consecutive user messages into one turn and the block otherwise reads as part of the user's text.
- Lookback: place the second message
cachePointon the latest human message before the tail. On a turn's first call that is the human message that opened the previous turn, whose prefix is already cached. This stays within the limit of 4 cache points. - C: replay reasoning to Bedrock as
reasoningContentblocks when a signature is available, and keep the signature when the run's content is persisted (today only Vertex signatures survive). Until signatures are persisted, the two changes above recover most of the loss, because the matching prefix ends right before the first assistant step.
Impact
Measured over one week on a production deployment: at least 44% of the turns sent less than 5 minutes after the previous one (352 of 805) read only the system prompt on their first call. Making the second checkpoint reachable plus moving the dynamic block brought the first call of each follow-up turn back to reading everything up to the previous turn's opening message.
Related in LibreChat
LibreChat persists think content parts without the provider signature, so even a converter that replays reasoning has nothing signed to send on the next turn. If preferred, this part can be tracked as a separate LibreChat issue.
Activity
Haarris commentedon Oct 21, 2026
Thanks for the detailed write-up. From what I can see, #619 covers the reasoning replay (C) and #613 keeps the conversation-level context at a fixed system position, but nothing open yet changes the message checkpoints, so the lookback part is still unaddressed.
Are you planning a PR for the second message
cachePoint? If not, I'm happy to open one: keep the tail checkpoint, add one on the latest human message before the tail, and only when it fits under Bedrock's limit of 4 together with the system and tool checkpoints (so it stays valid if #613 adds a second system point).manoemco commentedon Oct 26, 2026
We're independently hitting this same pattern in production, on @librechat/agents 3.2.57 via LibreChat's Bedrock agents endpoint, model us.anthropic.claude-sonnet-4-6 (cross-region inference profile).
Clearest single data point: within one assistant turn's internal tool-calling loop (same response messageId, many sequential internal LLM calls), one internal call got a clean, cheap cache hit, cacheReadInputTokens: 900441, cacheCreationInputTokens: 6. The very next internal call in that same loop didn't build on that checkpoint at all: it read only 70821 tokens (an older, smaller checkpoint) and paid a fresh write of 809192 tokens. The 900K checkpoint established one step earlier was never reused.
Across one longer exploratory conversation (48 internal LLM calls over several real turns, ~3.5 hours, heavy sequential tool use against a SQL-backed MCP server), cache-write tokens dominated cache-read tokens throughout rather than reads growing to dominate as the loop progressed (avg per call: ~26K fresh input, ~87K cache read, ~225K cache write). This matches causes 2 and 3 from the original report dynamic-instruction placement after the Bedrock system cache point, and the tail checkpoint's limited lookback once a turn accumulates several tool steps.
Real-world cost impact: this single conversation accounted for the large majority of our account's total Bedrock token volume for that day. For anyone running tool-heavy Bedrock+Claude agents, this isn't just a caching inefficiency , it's a meaningful, hard-to-notice cost multiplier.
We reviewed upgrading to 4.1.1 but see PRs #613 and #619, which look like they'd directly address this , are both still open. We'll upgrade and opt into the new reasoning-replay/conversation-instructions flags once either merges. Happy to share more of our token-count data if it helps with repro or review priority.
Haarris commentedon Oct 26, 2026
Thanks for sharing real numbers, that's really useful.
The 900K read followed by a 70K read and an 809K write inside one turn is interesting. Each tool step normally adds just the tool call and its result, which is well inside Bedrock's ~20 block lookback, so the next call should still reach the 900K checkpoint. Dropping back to ~70K (roughly system + tools) and re-writing everything after it usually means something changed early in the prompt.
At ~900K tokens you're close to the context limit, so two things worth checking for that step:
If you also see misses on the first call of a new turn, #631 targets that case. It adds a second message cache point so a rebuilt turn can still reach the previous turn's cache. In my Bedrock tests (sonnet-4-6) the first call of turn 2 went from 0% to about 70% cache read.
Haarris commentedon Oct 26, 2026
Happy to dig into your token data if you're up for sharing it. The per-call breakdown around that 900K -> 70K step (number of messages and blocks before and after, and whether a summary or pruning happened) would show pretty quickly what changed. If you'd rather not post it publicly, you can reach me through my GitHub profile, I do this kind of Bedrock caching work.
manoemco commentedon Oct 26, 2026
Thanks for digging in. Checked both:
Pruning/summarization: No, the persisted context snapshot for that exact step shows summaryTokens: 0. Summarization did happen later in this same conversation (twice, at two subsequent turns), but not until well after this drop.
Bulk tool results: No, the message for that turn has only 3 content blocks (reasoning, one tool call, final text). Single tool round-trip, not a batch of results landing at once.
For context, the full snapshot at that step: maxContextTokens: 889200 (model's native ceiling , we hadn't set an app-level cap yet), instructionTokens: 61363, systemMessageTokens: 11969, toolSchemaTokens: 634811 (196 MCP tools attached to this agent), messageCount: 70, messageTokens: 818651.
Since neither of those explains it, and the actual structural delta between the 900K-read call and the 809K-write call looks minimal (one tool call's worth), this looks like it's squarely the lookback/checkpoint-placement issue rather than something on our end. We also found #631 independently , looks like it targets cross-turn cache misses specifically ("first call of a new turn"); does it also cover a checkpoint getting dropped within a single turn's tool-calling loop, like what we're seeing here? Happy to share the raw transaction/context-snapshot JSON if useful.
Haarris commentedon Oct 26, 2026
Thanks, that rules out the obvious ones. One number stands out: the bad call read 70,821 tokens, and your instructions + system message come to 73,332. So the cache matched only up to the end of the system part, and everything after it, the 196 tool definitions (~635K) and the messages, was written again. With a 3-block delta, lookback can't be the cause here. Something before the messages changed between the two calls, most likely the tool list: MCP tools coming back in a different order, a tool description with something dynamic in it, or a tool loaded mid-turn.
To your question: no, #631 doesn't cover this. It's for the first call of a new turn, not a checkpoint lost inside one.
Yes please on the JSON. If you can share the two request bodies (the 900K-read call and the 809K-write call), a diff of the tools and system sections should show exactly what changed. Happy to look at it, email is on my GitHub profile if you'd rather not post it here.
manoemco commentedon Oct 26, 2026
Ran a controlled test , single tool call within one turn, synthetic data (nonexistent item IDs). The two consecutive calls came back byte-identical: same system prompt, same 127 tools, same order, nothing changed. Cache should have (and likely did) hit cleanly here.
So this simple case doesn't reproduce the drop, the original incident only showed it under much heavier conditions (a long multi-hour session, large accumulated context, possibly parallel tool calls in one turn, which this test didn't trigger). We're hesitant to force a bigger synthetic repro since it starts recreating the same scale/cost issue we were trying to diagnose in the first place, and we'd rather not run that against production again deliberately.
Does the bug only manifest under scale/parallel-tool-call conditions specifically, or is there a cheaper way to trigger the specific divergence (tool order/dynamic content) you suspect, without needing a huge context?