[tinker] Encode forwarded sample results to proto once and serve them as-is - #15
Closed
avigyabb wants to merge 2 commits into
Closed
[tinker] Encode forwarded sample results to proto once and serve them as-is#15avigyabb wants to merge 2 commits into
avigyabb wants to merge 2 commits into
Conversation
This was referenced Sep 4, 2026
… as-is Long-output rollouts made the per-result payload work the API server's main cost. For a 32k-token result (356KB JSON) the forwarding path spent 4ms (orjson decode, pydantic validate, pydantic JSON dump) and the proto path the SDK >= 0.25 uses on retrieve_future spent another 7ms (stdlib json.loads plus proto build) inside a single global lock, capping proto delivery near 150 results/s regardless of concurrency; the proto build holds the GIL, so the thread hop bought nothing. - The forwarding client decodes the vLLM body once and encodes straight to SampleResponse wire bytes (serialize_sample_output, shared with the validated path and pinned to it byte for byte by tests). No pydantic model or JSON text is built for the result. - ExternalFutureStore keeps the proto bytes (8 bytes/token, 26% smaller than the JSON text); retrieve_future passes them through for proto clients and derives JSON lazily, cached, for pre-proto clients (sample_output_json_from_proto). Results stored as JSON (DB path, errors) keep the existing encode-in-thread path, now cached per entry. - Pending entries no longer retain the request body (never read back on this path; ~100KB per entry for long prompts). - Store TTLs are EngineConfig fields (external_future_retrieved_ttl_sec, external_future_completed_ttl_sec): retention after delivery is the dominant memory term, roughly completion rate x result size x window. 512 concurrent 32k-token results, proto client: server CPU 7.5s -> 3.5s, 73 -> 167 results/s. 131072 requests with 8k-token results, 2048-way engine queueing and SDK-style 45s re-polls: 131072/131072, 0 failures, 267/s, peak RSS 8.9GB. 32768 requests with 32k-token results: 32768/32768, 163/s, 9.6GB. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Avi Basnet <avigyabb@stanford.edu> (cherry picked from commit ac48818) Signed-off-by: Avi Basnet <avigyabb@stanford.edu>
avigyabb
force-pushed
the
avi/stack-4-server-knobs
branch
from
September 4, 2026 01:24
2f94455 to
27c838e
Compare
avigyabb
force-pushed
the
avi/stack-5-proto-once
branch
from
September 4, 2026 01:24
a76b517 to
94b5591
Compare
avigyabb
commented
Sep 4, 2026
| class ExternalFuture: | ||
| request_id: int | ||
| model_id: str | None | ||
| request_data: dict |
Owner
Author
There was a problem hiding this comment.
we could retain a hashed version of this for idempotency, see - hershg#1
Removed comments explaining retention and population of result data in ExternalFuture class.
avigyabb
marked this pull request as ready for review
September 4, 2026 18:03
Owner
Author
|
Moved upstream: NovaSky-AI#2164 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 5/7. Long-output rollouts made per-result payload work the API server's main cost.
Before. For a 32k-token result (356 KB JSON) the forwarding path spent 4 ms (orjson decode, pydantic validate, pydantic JSON dump) and the proto path the SDK >= 0.25 uses on
retrieve_futurespent another 7 ms (stdlibjson.loads+ proto build) inside a single global lock, capping proto delivery near 150 results/s regardless of concurrency. The proto build holds the GIL, so the thread hop bought nothing.Change.
SampleResponsewire bytes (serialize_sample_output, shared with the validated path and pinned to it byte-for-byte by tests). No pydantic model or JSON text is built for the result.ExternalFutureStorekeeps the proto bytes (8 bytes/token, 26% smaller than the JSON text);retrieve_futurepasses them through for proto clients and derives JSON lazily, cached, for pre-0.25 clients (sample_output_json_from_proto, float32 logprobs). Results stored as JSON (DB path, errors) keep the encode-in-thread path, now cached per entry.EngineConfigfields (external_future_retrieved_ttl_sec,external_future_completed_ttl_sec): retention after delivery is the dominant memory term, roughly completion rate x result size x window.Results (proto client, 2048 cap): 512 concurrent 32k-token results 7.5 s -> 3.5 s server CPU, 73 -> 167 results/s. 131072 requests with 8k-token results, 5 s engine queueing and SDK-style 45 s re-polls: 131072/131072, 0 failures, 267/s, 8.9 GB peak RSS. 32768 x 32k tokens: 32768/32768, 163/s, 9.6 GB.
Stack
🤖 Generated with Claude Code