[tinker] Long-output results: pysimdjson decode, burst-survival settings, harness fidelity (131072 x 212k tokens)

[avigyabb]

Stacked on #9. With results the size of Chuck's repro (262k tokens, 2.8 MB of JSON per result), decoding the vLLM body was 72% of the API server's CPU even after #9's proto-once path: orjson.loads builds one Python object per token and logprob (22 ms), then numpy converts the lists (16 ms). This PR decodes the two numeric arrays straight into int32/float32 buffers.

Change

• skyrl/tinker/extra/completion_decode.py: CompletionDecoder parses /v1/completions bodies with pysimdjson and pulls token_ids / token_logprobs out as raw numeric buffers (Array.as_buffer), so a 262k-token result decodes in ~6 ms with no per-token Python objects. Falls back to orjson when pysimdjson is not installed, and per array when an array is not purely numeric (a null logprob). Zero-fill semantics for missing/null logprobs are unchanged from the forwarding client.
• SkyRLTrainInferenceForwardingClient._forward uses it; the decoded arrays feed serialize_sample_output directly (numpy in, no copy).
• pysimdjson added to the tinker extra (uv.lock +46 lines).
• Harness: --retrieved-ttl passthrough so long-output runs can bound retention memory.

Decode alternatives measured on one 262k-token body: current 38.6 ms; np.fromstring on extracted substrings 28.9 ms; pysimdjson 6.2 ms.

Results (proto client, 262k-token results, 512 outbound cap)

Run	Completed	Throughput	Server CPU	Peak RSS
#9, 512 concurrent	512/512	32/s	16.6 s (decode 12.0 s)	1.8 GB
This PR, 512 concurrent	512/512	89/s	6.25 s	1.6 GB
This PR, 8192 with 5 s engine queueing, 45 s re-polls, 60 s retrieved TTL	8192/8192, 0 failures	79/s (~230 MB/s of results)	saturated	15.9 GB

Memory at this size is retention: completion rate x 2.1 MB proto x external_future_retrieved_ttl_sec. The default 300 s would hold ~50 GB at 80 results/s, so operators running 262k-token rollouts should set the TTL to 60-120 s (the SDK re-polls a lost response within 45 s + 30 s backoff).

Tests: tests/tinker/test_completion_decode.py covers both backends, nulls, empty arrays, prompt-logprob passthrough, non-JSON bodies, parser reuse, and 200k-element parity between the fast and fallback paths.

Two more commits: surviving completion bursts, and harness fidelity

f3a58f1b server: with 131072 outstanding samples and 212k-token results, each 2048-result completion burst kept the loop busy ~16 s; uvicorn's 5 s keep-alive then closed every idle client connection, all clients reconnected at once and the 2048-entry accept backlog overflowed (109k of 131072 requests refused at TCP level). uvicorn.run now uses backlog=SKYRL_HTTP_CONNECTION_LIMIT (as on Chuck's branch; effective value capped by net.core.somaxconn, raise it to match) and timeout_keep_alive=75. The forwarding connector gets a 60 s connect timeout and Happy Eyeballs off: a wave of 10 s connect timeouts toward a saturated router produced ~800 uvloop File descriptor N is used by transport errors out of aiohappyeyeballs' cancelled sock_connect, failing unrelated forwards.

e295b282 harness: --max-outstanding emulates the SDK's sample_max_concurrent_requests; asample/retrieve failures retry 16x with backoff like the SDK; --router-workers N runs the fake router across processes (one aiohttp process saturates at ~250 MB/s of 2.3 MB bodies); --retrieved-ttl.

131072 requests x 212k-token results (proto client, 2048 forwarding cap, 5 s engine queueing, 16384 in flight, 60 s retrieved TTL)

Attempt	Completed	Wall	Throughput	Peak RSS	Notes
uncapped outstanding, old settings	19070 / 131072	273 s (collapsed)		20.5 GB	accept-backlog overflow after keep-alive closures
capped, backlog+keep-alive	130149 forwarded, run cut at 70 min			19.6 GB	1972 forwards failed: router connect timeouts + fd-reuse errors
capped, hardened connector, 4 router workers	130917 / 131072	1144 s	114 results/s (~190 MB/s)	20.4 GB	0 forwarding errors; 155 asample responses took >45 s (the SDK retries these; the harness at the time did not)

Caveat on the last row: four router workers from a preceding smoke run were still bound to the port (fixed in e295b282), so part of the traffic saw 2 s instead of 5 s generation; the API-server measurements are unaffected.

Where the API server stands at this payload size: ~9 ms of CPU per result (decode + proto) plus ~2 ms of HTTP, so one process delivers 110-120 results/s of 212k-token output. Retention memory is completion rate x 1.7 MB x retrieved TTL. In production the vLLM API server serializing 2.3 MB of JSON per result is likely to saturate first.

Before / after vs upstream main (345ce86), same harness and SDK-shaped client

Scenario	main	#9 + #10
2048 small results, unlimited connections	71.6 s, 29/s	2.7 s, 747/s
512 x 32k-token results, proto	10.3 s, 50/s	3.1 s, 167/s
512 x 262k-token results, proto	83.8 s, 6.1/s	5.8 s, 89/s
32768 small, 5 s engine queueing	218 s, 148/s, 530 failures	89 s, 370/s, 0 failures
131072 small, queueing, 45 s re-polls	not feasible	366 s, 358/s, 0 failures
131072 x 8k-token, proto, queueing	not feasible	491 s, 267/s, 0 failures

And on pre-NovaSky-AI#2097 main (59d4daed, where Chuck saw 32x64 fail) with 262k results: 512 barrier -> 511/512 in 1441 s with QueuePool timeouts; 2048 with queueing -> 18/2048 in 39 min, 4218 QueuePool errors, the client saw "QueuePool limit of size 5 overflow 10 reached". The same 512 case here: 5.8 s.

Tests: tinker + utils CPU suite, 117 passed.

🤖 Generated with Claude Code

[avigyabb]

Superseded by the split stack (each PR shows only its own diff and retargets to main as the one below merges):

1. [tinker] Route uvicorn's access log to a plain handler instead of Rich #11
2. [tinker] Forward samples with aiohttp instead of httpx #12
3. [tinker] Keep an undelivered sample result alive for the SDK's retry #13
4. [tinker] Survive completion bursts at the socket layer (accept backlog, keep-alive) #14
5. [tinker] Encode forwarded sample results to proto once and serve them as-is #15
6. [tinker] Decode vLLM completion bodies straight into numpy with pysimdjson #16
7. [tinker] Load harness for the API server's sampling path at 131k concurrency #17