[tinker] Survive completion bursts at the socket layer (accept backlog, keep-alive) - #14
Closed
avigyabb wants to merge 1 commit into
Closed
[tinker] Survive completion bursts at the socket layer (accept backlog, keep-alive)#14avigyabb wants to merge 1 commit into
avigyabb wants to merge 1 commit into
Conversation
This was referenced Sep 4, 2026
uvicorn.run: backlog=SKYRL_HTTP_CONNECTION_LIMIT (50k; the effective value is capped by net.core.somaxconn, raise it to match) and timeout_keep_alive=75s. With 131072 outstanding samples and 212k-token results each 2048-result completion burst kept the event loop busy ~16s. uvicorn's 5s keep-alive then closed every idle client connection during the burst, every client reconnected at once, and the 2048-entry accept backlog overflowed, so the kernel refused 109k of 131072 requests. Connections now queue in the kernel while the loop is busy and idle ones survive a burst. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Avi Basnet <avigyabb@stanford.edu>
avigyabb
force-pushed
the
avi/stack-4-server-knobs
branch
from
September 4, 2026 01:24
2f94455 to
27c838e
Compare
avigyabb
changed the title
avigyabb
marked this pull request as ready for review
September 4, 2026 01:26
Owner
Author
|
Moved upstream: NovaSky-AI#2163 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 4/7.
uvicorn.run:backlog=SKYRL_HTTP_CONNECTION_LIMIT(50k, as on Chuck's branch; the effective value is capped bynet.core.somaxconn, raise it to match) andtimeout_keep_alive=75.With 131072 outstanding samples and 212k-token results, each 2048-result completion burst kept the event loop busy ~16 s; uvicorn's 5 s keep-alive then closed every idle client connection, all clients reconnected at once and the 2048-entry accept backlog overflowed, refusing 109k of 131072 requests. With these settings the same run completed 130917 of 131072 with zero forwarding errors and zero reconnects (earlier runs on the 5 s keep-alive showed hundreds to thousands).
Neither setting is needed for correctness: the SDK retries refused or dropped connections. They avoid the reconnect storm rather than fix a failure, and with the SDK's per-client in-flight cap the burst that overflowed the backlog does not occur. An earlier revision of this PR also exposed
sample_max_concurrent_requestsfromEngineConfigvia/client/config; that was dropped as unnecessary (its default equalled the SDK default, and no measured run depended on it).Stack
🤖 Generated with Claude Code