[tinker] Keep an undelivered sample result alive for the SDK's retry

[avigyabb]

Stack 3/7. Fixes the 128x128 404 Future not found seen against j316chuck#18.

Chain. The SDK polls retrieve_future with a 45 s client timeout and gives up; the result lands afterwards; the abandoned handler wakes, builds a response nobody receives (uvicorn drops the send to a dead client silently) and starts the short retrieved-TTL clock; the sweeper evicts the result 120 s later; the SDK's retry of the same request_id gets 404, which the SDK treats as fatal.

Fix. Start the retrieved clock only if request.is_disconnected() is false, and raise the retrieved TTL to 300 s so it outlasts the SDK's worst-case re-poll gap (45 s timeout + up to 30 s backoff, twice). tests/tinker/test_retrieve_future_lost_response.py reproduces the chain under a real uvicorn socket with shortened TTLs; it fails on main and passes here. A second test checks a delivered result still expires on the short clock, so memory stays bounded.

Alternative considered: j316chuck#19 drops the retrieved clock and keeps every result for 2048 s. That also fixes the 404 but retains ~35 minutes of results regardless of delivery; with long-output rollouts (hundreds of KB per result) that is tens of GB. Verified at scale: 131072 requests with 5 s engine queueing and 224k SDK-style abandoned polls completed with zero 404s.

Stack

1. [tinker] Route uvicorn's access log to a plain handler instead of Rich #11
2. [tinker] Forward samples with aiohttp instead of httpx #12
3. [tinker] Keep an undelivered sample result alive for the SDK's retry #13
4. [tinker] Survive completion bursts at the socket layer (accept backlog, keep-alive) #14
5. [tinker] Encode forwarded sample results to proto once and serve them as-is #15
6. [tinker] Decode vLLM completion bodies straight into numpy with pysimdjson #16
7. [tinker] Load harness for the API server's sampling path at 131k concurrency #17

🤖 Generated with Claude Code

[avigyabb]

Moved upstream: NovaSky-AI#2162