[tinker] Load harness for the API server's sampling path at 131k concurrency

[avigyabb]

Stack 7/7. The harness that produced every number in stacks 1-6.

skyrl/benchmarks/load_test_tinker_sampling.py isolates the Tinker API server (FastAPI/uvicorn, SQLite, in-memory future store, forwarding client) from the engine, CPU only: the real skyrl.tinker.api app under uvicorn configured as the non-colocated megatron server with the engine subprocess stubbed and the router URL seeded into EngineStateDB; a fake vLLM router (barrier or fixed-latency mode, optionally multi-process for multi-MB results); and a load client that behaves like the Tinker SDK (400 concurrent submits, sample_max_concurrent_requests cap, 45 s re-polls, 16 connection retries with backoff) or drives the real SDK (--client sdk). Workers bind distinct loopback source IPs so one IP's ~28k ephemeral ports do not cap the run. Reports completed/failed by class, latency percentiles, the router's peak in-flight count, and the server's peak RSS, fds and /healthz latency; TINKER_LOADTEST_PROFILE=<path> cProfiles the server process.

uv run --extra tinker python skyrl/benchmarks/load_test_tinker_sampling.py \
    --num-requests 131072 --forwarding-max-connections 2048

# realistic engine queueing, long outputs, SDK cap
uv run --extra tinker python skyrl/benchmarks/load_test_tinker_sampling.py \
    --num-requests 131072 --max-tokens 8192 --proto --forwarding-max-connections 2048 \
    --vllm-mode latency --max-num-seqs 2048 --gen-seconds 5 --max-outstanding 16384

# against a real server with the public SDK
uv run --extra tinker python skyrl/benchmarks/load_test_tinker_sampling.py \
    --role load --client sdk --url http://HOST:8000 --num-requests 4096

Structure follows Chuck Tang's SQLite QueuePool repro gist (https://gist.github.com/j316chuck/f44f35572ffb8584519d13b943f99ef8). Run against pre-NovaSky-AI#2097 main (59d4daed) with 262k-token results it reproduces his 32x64 collapse: 18 of 2048 completed in 39 min with 4218 QueuePool errors.

What it does not model: real generation time (production is far gentler on the server than the harness), engine and weight-sync stalls, training traffic on the same SQLite file, and a real network between SDK and server.

Stack

1. [tinker] Route uvicorn's access log to a plain handler instead of Rich #11
2. [tinker] Forward samples with aiohttp instead of httpx #12
3. [tinker] Keep an undelivered sample result alive for the SDK's retry #13
4. [tinker] Survive completion bursts at the socket layer (accept backlog, keep-alive) #14
5. [tinker] Encode forwarded sample results to proto once and serve them as-is #15
6. [tinker] Decode vLLM completion bodies straight into numpy with pysimdjson #16
7. [tinker] Load harness for the API server's sampling path at 131k concurrency #17

🤖 Generated with Claude Code

[avigyabb]

Moved upstream: NovaSky-AI#2166