Skip to content

[tinker] Raise SQLite busy timeout under rollout load - #2124

Open
j316chuck wants to merge 1 commit into
NovaSky-AI:mainfrom
j316chuck:chuck-codex/nemotron-sqlite-busy-timeout
Open

[tinker] Raise SQLite busy timeout under rollout load#2124
j316chuck wants to merge 1 commit into
NovaSky-AI:mainfrom
j316chuck:chuck-codex/nemotron-sqlite-busy-timeout

Conversation

@j316chuck

@j316chuck j316chuck commented Sep 1, 2026

Copy link
Copy Markdown
Contributor
  • Raise SQLite's busy timeout from 30 seconds to 300 seconds for sustained rollout write contention.\n- Isolates the lock-timeout patch removed from Trajectory PR #4385; DB-pool sizing remains in a separate PR.\n\n## Testing\n\nbash\nuv run --no-sync ruff check skyrl/tinker/db_models.py\n\n\nRuff passed. The focused DB suite could not collect in the shared environment because its tinker.proto test dependency is absent.

@j316chuck
j316chuck marked this pull request as ready for review September 1, 2026 15:31

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request increases the SQLite busy_timeout from 30 seconds to 300 seconds in skyrl/tinker/db_models.py. Feedback suggests that a 5-minute timeout is excessively high and risks thread pool starvation under heavy write contention. It is recommended to use a more reasonable timeout, such as 60 seconds, combined with application-level retries.

Comment thread skyrl/tinker/db_models.py
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA journal_mode=WAL")
cursor.execute("PRAGMA busy_timeout=30000")
cursor.execute("PRAGMA busy_timeout=300000")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Increasing the SQLite busy_timeout to 300 seconds (5 minutes) is extremely high and poses a significant risk of thread pool starvation. Since this application uses aiosqlite (which runs blocking SQLite operations in a thread pool), blocking a database thread for up to 5 minutes under heavy write contention can quickly exhaust the thread pool. This will freeze all other database operations (including reads) and potentially cause health checks or other asynchronous tasks to time out.

Consider using a more reasonable timeout (e.g., 60 seconds) combined with application-level retries with exponential backoff, or optimizing transaction boundaries to keep write locks as short as possible.

Suggested change
cursor.execute("PRAGMA busy_timeout=300000")
cursor.execute("PRAGMA busy_timeout=60000")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants