Document embedding latency as a retrieval-side design decision

[jasperblues]

Docs only. Records a finding from chasing per-turn chat latency downstream, so the next person doesn't have to measure it again.

What it says

Every vector retrieval embeds the query before it can search. When the query is a user's message and the result is needed to build a prompt, that call is on the critical path — nothing can be sent to the model until it returns.

Measured on text-embedding-3-small over ~3000 propositions:

embed + vector search	300–355ms (occasional ~1.3s outlier)
propositions returned for a greeting	0
projection + provenance	~0ms

Two things worth reading carefully: the cost is the embedding round trip, not the vector search — the graph side is fast and with no hits there's nothing to project. And the retrieval was correct; a greeting genuinely matches nothing. The expense isn't retrieving, it's finding out there's nothing to retrieve, paid on every turn.

Why it belongs in dice

The asymmetry is a property of the design, not of one deployment:

• Ingestion-side embedding is throughput-bound, batched, off the critical path. Hosted model quality is worth having.
• Retrieval-side embedding is latency-bound, one short string, blocking a user. Local (ONNX) wins.

Nothing in retrieval assumes both use the same model — the embedding service is injected. The doc notes the constraint from durable-storage.md that query and stored vectors must share model and dimension, so this only works as the same model served locally, not a different one.

Also records what does not fix it

Both are the intuitive first answers and neither survives measurement, so they're written down explicitly:

• Async/parallel retrieval — the result is required to build the prompt, so it's on the critical path by construction. Measured case had ~80ms of overlappable work against a 330ms retrieval.
• Deadline-bounding — caps the worst case but makes prompt content depend on network jitter, trading determinism and any memory-dependent eval for latency.

🤖 Generated with Claude Code