[tinker] Keep an undelivered sample result alive for the SDK's retry

[avigyabb]

Stack 3/7. Fixes the 128x128 404 Future not found seen against j316chuck#18.

Chain. The SDK polls retrieve_future with a 45 s client timeout and gives up; the result lands afterwards; the abandoned handler wakes, builds a response nobody receives (uvicorn drops the send to a dead client silently) and starts the short retrieved-TTL clock; the sweeper evicts the result 120 s later; the SDK's retry of the same request_id gets 404, which the SDK treats as fatal.

Fix. Start the retrieved clock only if request.is_disconnected() is false, and raise the retrieved TTL to 300 s so it outlasts the SDK's worst-case re-poll gap (45 s timeout + up to 30 s backoff, twice). tests/tinker/test_retrieve_future_lost_response.py reproduces the chain under a real uvicorn socket with shortened TTLs; it fails on main and passes here. A second test checks a delivered result still expires on the short clock, so memory stays bounded.

Alternative considered: j316chuck#19 drops the retrieved clock and keeps every result for 2048 s. That also fixes the 404 but retains ~35 minutes of results regardless of delivery; with long-output rollouts (hundreds of KB per result) that is tens of GB. Verified at scale: 131072 requests with 5 s engine queueing and 224k SDK-style abandoned polls completed with zero 404s.

Stack (each PR retargets to main as the one below merges)

1. [tinker] Route uvicorn's access log to a plain handler instead of Rich #2160
2. [tinker] Forward samples with aiohttp instead of httpx #2161
3. [tinker] Keep an undelivered sample result alive for the SDK's retry #2162
4. [tinker] Survive completion bursts at the socket layer (accept backlog, keep-alive) #2163
5. [tinker] Encode forwarded sample results to proto once and serve them as-is #2164
6. [tinker] Decode vLLM completion bodies straight into numpy with pysimdjson #2165
7. [tinker] Load harness for the API server's sampling path at 131k concurrency #2166

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes async polling/delivery semantics and HTTP server tuning on the hot retrieve_future path; behavior is covered by new integration tests but affects SDK retry reliability under load.

Overview
Fixes fatal 404 Future not found when the SDK abandons a long retrieve_future poll (45s client timeout) and retries the same request_id after the result is ready.

retrieve_future now calls mark_retrieved (starting the post-delivery eviction clock) only when the client is still connected (not await req.is_disconnected()). If the handler finishes building a response after the client disconnected, the short retrieved TTL no longer starts, so the in-memory store keeps the result for a real retry.

ExternalFutureStore raises _RETRIEVED_TTL_SECONDS from 120s to 300s so delivered results still get a grace window that covers worst-case SDK re-poll gaps (timeout + backoff, twice).

Uvicorn startup sets timeout_keep_alive=75 (vs 5s default) and backlog=SKYRL_HTTP_CONNECTION_LIMIT to reduce idle disconnects and accept-queue overflows during completion bursts.

Adds test_retrieve_future_lost_response.py (real socket on Linux) plus test stubs for is_disconnected on existing API tests.

^{Reviewed by Cursor Bugbot for commit 36188da. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge with no concrete blocking or independently actionable non-blocking issue established.

The connection-state gate preserves undelivered results for retry, the longer retrieved TTL covers the documented retry interval, and the tests exercise both abandoned and successfully delivered response lifecycles.

Important Files Changed

Filename	Overview
skyrl/tinker/api.py	Gates the retrieved-result expiration clock on the polling client still being connected.
skyrl/tinker/external_future_store.py	Extends the post-retrieval retention period to cover the documented SDK retry cadence.
tests/tinker/test_external_future_store.py	Updates in-memory request stubs to provide the new asynchronous connection-state check.
tests/tinker/test_retrieve_future_lost_response.py	Adds real-uvicorn regression coverage for abandoned polling connections and normal delivered-result expiration.

Sequence Diagram

sequenceDiagram
    participant SDK
    participant API as retrieve_future
    participant Store as ExternalFutureStore
    SDK->>API: Poll request_id
    API->>Store: Wait for result
    SDK--xAPI: Client timeout / disconnect
    Store-->>API: Result completes
    API->>API: is_disconnected()
    alt Client disconnected
        API-->>Store: Keep completed TTL
    else Client connected
        API->>Store: mark_retrieved()
        API-->>SDK: Return result
    end
    SDK->>API: Retry same request_id
    API->>Store: Retrieve retained result
    Store-->>SDK: Result

Loading

_{Reviews (1): Last reviewed commit: "[tinker] Keep an undelivered sample resu..." | Re-trigger Greptile}

[gemini-code-assist]

Code Review

This pull request addresses an issue where a client-side timeout on a poll causes the server to prematurely start the short retrieved TTL clock, leading to a 404 error on subsequent retries. To resolve this, the server now checks if the client is still connected before marking a future as retrieved. Additionally, the retrieved TTL has been increased from 120 to 300 seconds to safely cover client backoff periods, and comprehensive integration tests have been added to verify these changes. No review comments were provided, so there is no additional feedback.