Skip to content

fix(export): stream database dump in batches for large DBs #59#266

Open
FishEnjoyer2025 wants to merge 1 commit into
outerbase:mainfrom
FishEnjoyer2025:feat/streaming-database-dump
Open

fix(export): stream database dump in batches for large DBs #59#266
FishEnjoyer2025 wants to merge 1 commit into
outerbase:mainfrom
FishEnjoyer2025:feat/streaming-database-dump

Conversation

@FishEnjoyer2025
Copy link
Copy Markdown

Fixes #59 — database dumps failing on large databases.

Problem

dumpDatabaseRoute assembled the entire dump in a single in-memory string and read each
table with one unbounded SELECT *. On large databases this exhausts memory and/or
exceeds the 30s request window.

Fix

  • Stream the response via a ReadableStream, so bytes flow to the client as they're
    produced — the connection stays active instead of buffering one giant body, which keeps
    the request from hitting the 30s wall.
  • Read each table in bounded batches (LIMIT/OFFSET, default 1000 rows), so memory
    usage stays roughly constant regardless of table size.
  • The initial table-list query stays up front so an early failure still returns a clean
    500 (rather than a half-streamed 200). batchSize is an optional parameter for tuning.

Tests

All existing dump.test.ts cases still pass, plus a new one verifying that a table larger
than the batch size is streamed across multiple OFFSET queries. 6/6 passing.

Notes

This solves the common case (constant memory + no 30s timeout) with no intermediate
storage. For truly massive (10GB+) dumps that could exceed a Durable Object's execution
budget even while streaming, a follow-up could spool to R2 and return a download URL —
happy to do that as a separate PR if you'd prefer that direction.

/claim #59

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant