[tinker] Forward samples with aiohttp instead of httpx

[avigyabb]

Stack 2/7.

Why. httpcore's connection pool rescans every pooled connection and queued request on each event, so the forwarding client's per-request CPU grows with the number of in-flight samples: 14 ms at 512, 27 ms at 2048, 38 ms at 4096 (standalone benchmark against an instant fake router; the profile shows 27.5M is_idle calls for 512 requests). At 2048 in flight the API server needed 119 s to forward 2048 instant requests. aiohttp's connector is flat at 0.34 ms.

Change. SkyRLTrainInferenceForwardingClient uses one aiohttp session:

• connector limit = forwarding_inference_max_connections (0 = unlimited); sock_read = forwarding_inference_timeout_sec; no total deadline, so requests queued behind the engine never hit the old 300 s pool timeout;
• 60 s connect timeout (a saturated router takes tens of seconds to accept; that is queueing, not failure) and Happy Eyeballs off: a wave of cancelled connects under uvloop produced ~800 File descriptor N is used by transport errors from aiohappyeyeballs' sock_connect, failing unrelated forwards;
• connect-phase errors and 5xx rejections (TransientInferenceError) are retried once after refreshing the proxy URL; read failures stay final (fix(tinker): bound inference forwarding retries NovaSky-AI/SkyRL#2118 stance) since vLLM may still be executing the request;
• forwarding_inference_timeout_sec default 300 s -> 2048 s: with unlimited connections a large burst waits inside vLLM's queue and 128x128 bursts exceed 300 s there.

Results (same harness and SDK-shaped client, upstream main vs this stack): 2048 small results unlimited connections 71.6 s / 29 per s -> 2.7 s / 747 per s; 32768 with 5 s engine queueing 218 s with 530 failures -> 89 s, 0 failures.

Note: unlimited outbound connections cannot exceed ~28k (one source IP's ephemeral ports); set --forwarding-inference-max-connections near engine capacity for very large bursts.

Stack

1. [tinker] Route uvicorn's access log to a plain handler instead of Rich #11
2. [tinker] Forward samples with aiohttp instead of httpx #12
3. [tinker] Keep an undelivered sample result alive for the SDK's retry #13
4. [tinker] Survive completion bursts at the socket layer (accept backlog, keep-alive) #14
5. [tinker] Encode forwarded sample results to proto once and serve them as-is #15
6. [tinker] Decode vLLM completion bodies straight into numpy with pysimdjson #16
7. [tinker] Load harness for the API server's sampling path at 131k concurrency #17

🤖 Generated with Claude Code

[avigyabb]

this runs in the uvicorn event loop via FastAPI asample()

[avigyabb]

Moved upstream: NovaSky-AI#2161