fix(tinker): align forwarding pool deadline - #2140
Conversation
Signed-off-by: Hersh Godse <hersh@trajectory.ai>
There was a problem hiding this comment.
Code Review
This pull request updates the httpx client configuration in skyrl_train_inference_forwarding.py to use the configured forwarding_inference_timeout_sec for the pool timeout instead of a hardcoded value of 300.0. The corresponding unit tests in test_inference_forwarding_config.py have been updated and renamed to reflect this change. There are no review comments, and I have no additional feedback to provide.
|
CI follow-up: code quality, gym, train, and the full SkyRL CPU suite pass on the final commit. The GPU workflow failed before submitting any remote job because the fork context has no Anyscale credentials; it did not reach GPU setup or tests. The candidate module was also loaded against the exact deployed runtime and confirmed connect/read/write/pool timeouts of 10/1800/300/1800 seconds. The unchanged 512-request B300 replay is in progress. |
Testing
The updated timeout-wiring assertion and local static checks pass; upstream CI provides the clean-environment suite.
Note
Low Risk
Narrow httpx timeout wiring change for inference forwarding; no auth or data-path logic, only avoids failing earlier than the configured inference deadline when the connection pool is saturated.
Overview
Fixes premature pool timeouts when forwarded inference requests wait for a free HTTP connection under high concurrency.
The forwarding client’s
httpx.Timeoutpool deadline was hard-coded to 300s while read already usedforwarding_inference_timeout_sec. Pool is now set to the same configured value, so raising--forwarding-inference-timeout-sec/SKYRL_FORWARDING_INFERENCE_TIMEOUT_SECapplies while waiting for a connection as well as while reading the vLLM response.Tests were updated to assert read and pool both match the configured timeout (e.g. 1800s).
Reviewed by Cursor Bugbot for commit e7c9f42. Bugbot is set up for automated code reviews on this repo. Configure here.