Skip to content

[tinker] Size the API database pool for rollout bursts - #2123

Open
j316chuck wants to merge 1 commit into
NovaSky-AI:mainfrom
j316chuck:chuck-codex/nemotron-db-pool
Open

[tinker] Size the API database pool for rollout bursts#2123
j316chuck wants to merge 1 commit into
NovaSky-AI:mainfrom
j316chuck:chuck-codex/nemotron-db-pool

Conversation

@j316chuck

@j316chuck j316chuck commented Sep 1, 2026

Copy link
Copy Markdown
Contributor
  • Raise the Tinker API SQLAlchemy pool to 50 connections with 100 overflow and a 120-second checkout timeout.\n- Isolates the DB-pool patch removed from Trajectory PR #4385; SQLite lock timing remains in a separate PR.\n\n## Testing\n\nbash\nuv run --no-sync ruff check skyrl/tinker/api.py\n\n\nRuff passed. The focused DB suite could not collect in the shared environment because its tinker.proto test dependency is absent.

Note

Low Risk
Single startup configuration change with no query or auth logic; main operational risk is higher peak DB connection use under burst load.

Overview
Sizes the Tinker API’s async SQLAlchemy pool so short-lived rollout traffic (many concurrent retrieve_future waiters and per-request AsyncSession checkouts) is less likely to exhaust connections or fail while waiting for a slot.

On startup in lifespan, create_async_engine now uses pool_size=50, max_overflow=100, and pool_timeout=120 seconds instead of SQLAlchemy’s smaller defaults.

Reviewed by Cursor Bugbot for commit cd03551. Bugbot is set up for automated code reviews on this repo. Configure here.

@j316chuck
j316chuck marked this pull request as ready for review September 1, 2026 15:31

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the database engine creation in skyrl/tinker/api.py to configure connection pool parameters (pool_size, max_overflow, and pool_timeout). The reviewer noted that applying these parameters directly can cause a TypeError with in-memory SQLite databases and lead to database lock contention for file-based SQLite. They recommended conditionally applying these parameters only when the database backend is not SQLite.

Comment thread skyrl/tinker/api.py
Comment on lines +236 to +238
app.state.db_engine = create_async_engine(
db_url, echo=False, pool_size=50, max_overflow=100, pool_timeout=120
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Passing pool_size, max_overflow, and pool_timeout directly to create_async_engine introduces two major issues:

  1. TypeError with In-Memory SQLite: When using an in-memory SQLite database (common in test suites), SQLAlchemy defaults to using StaticPool. StaticPool does not accept these pool configuration arguments, which will cause a TypeError and crash the application on startup.
  2. Database Lock Contention: For file-based SQLite (the default database configuration), a pool size of 50 with 100 overflow (up to 150 connections) is highly discouraged. SQLite only supports a single concurrent writer, and having so many concurrent connections will lead to severe database lock contention (database is locked / SQLITE_BUSY errors), even with WAL mode enabled.

To resolve this, we should conditionally apply these pool parameters only when the database backend is not SQLite (e.g., PostgreSQL).

Suggested change
app.state.db_engine = create_async_engine(
db_url, echo=False, pool_size=50, max_overflow=100, pool_timeout=120
)
engine_kwargs = {"echo": False}
if not db_url.startswith("sqlite"):
engine_kwargs.update(
pool_size=50,
max_overflow=100,
pool_timeout=120,
)
app.state.db_engine = create_async_engine(db_url, **engine_kwargs)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants