Repository navigation
Add an opt-in setting that lets the API Worker apply its own schema - #59
Open
MendyLanda wants to merge 2 commits into
Open
MendyLanda wants to merge 2 commits into
MendyLanda wants to merge 2 commits into
Conversation
Self-hosters and one-click installers such as Appflare deploy new versions without a checkout or a direct database URL, so they can't run `pnpm migrate` before each update. With MAIL_AUTO_MIGRATE_ENABLED=true, each isolate checks a schema marker in mail.schema_migrations once, before the first event that uses the database. If the marker is missing, it runs migrate() through Hyperdrive in one transaction under pg_advisory_xact_lock, without changing role. A failure rolls back and fails the event, and the next event retries. /healthz skips the check. migrate() records the marker, a hash of the migration list, so `pnpm migrate` and the Worker agree. The setting is off by default, so the hosted deployment keeps applying reviewed SQL by hand.
|
Author
|
@michaelshimeles this small addition will allow this app to be installed via appflare |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Self-hosters and one-click installers such as Appflare deploy new versions of the API Worker without a checkout or a direct database URL, so they can't run
pnpm migratebefore each update. This adds an opt-in Worker var,MAIL_AUTO_MIGRATE_ENABLED, that lets the Worker bring its own schema up to date. It is off by default, so the hosted service keeps applying reviewed SQL by hand and behaves exactly as it does today.Proposed in #58.
migrate()now records one marker row,schema-<hash>, inmail.schema_migrations. The hash covers the migration list, so any edit to a migration changes the marker.pnpm migrateand the native migration script callmigrate(), so they record the same row. The schema itself does not change.ensureSchema()indatabase.tslooks up the marker with one query. A fresh database has nomail.schema_migrationstable yet, and it treats that error,42P01, as out of date. When the marker is missing, it opens one transaction, takespg_advisory_xact_lockon a fixed key, checks the marker again, and runsmigrate(). A failure rolls the whole transaction back. Every statement runs inside that one transaction on one connection, with no session settings or session locks, so it works with Hyperdrive's transaction pooling. It never runsset local role postgres, which only works with PlanetScale's migration logins.worker.tskeeps oneensureSchema()promise per isolate and awaits it before the fetch, email, queue, and scheduled handlers touch the database. A rejection clears it so the next event tries again. The event that starts the migration also passes it towaitUntil, so the migration finishes even if that event ends first and later events never wait on a dead request.internal_errorwithtransient: truethat a database error gets. Email throws "Email could not be stored", as for a storage failure. Queue retries the whole batch after 60 seconds, like the existing per-message database retries. Scheduled throws, so the next cron run tries again./healthzskips the check. The database guide says it does not query the database, and I don't think an uptime probe should start or wait on a migration. The cron runs every minute, so a deploy with no traffic still migrates within a minute.trueorfalsemakes the Worker report that it is not configured.How I tested it
I ran everything with Node 24.19.0 and pnpm 10.15.1.
pnpm test:postgresran against a disposablepostgres:18-bookwormcontainer on127.0.0.1:5435, as in the database guide.pnpm buildpnpm checkpnpm testpnpm source:checkpnpm test:postgres(when SQL changed)test/auto-migrate.test.tsruns on PGlite inpnpm testand on PostgreSQL inpnpm test:postgres. It calls the Worker's real handlers and maps the Hyperdrive connection string to a test database. It checks that:false, a request never touchesmail.schema_migrations, and an invalid value reportsnot_configured;true,/healthzmakes no query, and an empty database ends up with the same tables, indexes, functions, and markers asmigrate()before the first request returns 200;ensureSchema()calls on an empty database return onemigratedand threecurrent, and six concurrent requests in one isolate share one transaction;mailobjects behind, and the next call finishes it.test/postgres-runtime.test.tsgains a workerd run through Wrangler's local Hyperdrive binding. The Hyperdrive user owns the database but can't becomepostgres, as on Neon or Supabase. Eight concurrent cold requests against an empty database all return 200, the mail objects belong to that user, and one schema marker is recorded. The existingpnpm migratetest now also checks thatensureSchema()returnscurrentafterward, so the CLI and the Worker agree on the marker.I also removed pieces of the change to see the tests catch it. With only the advisory lock removed, the concurrent test fails on PostgreSQL 18 in 3 of 3 runs with
duplicate key value violates unique constraint "pg_namespace_nspname_index". Removing the recheck under the lock, the code that clears a failed promise, or the gate itself also fails the matching tests on PGlite.Test doubles
The new tests make no provider calls. The existing doubles still apply. Cloudflare sending and routing return canned responses, the PGlite tests use a stub R2 binding and a fake queue batch, and the workerd test uses Wrangler's local R2. Wrangler's local Hyperdrive binding doesn't pool connections, so these tests don't prove behavior under deployed Hyperdrive. That's why the migration only uses transaction-scoped features.
Risks
migrate()writes one extra row tomail.schema_migrations.alter table ... add column if not existslocks the table even when the column exists, so requests can wait briefly. That's the same cost as runningpnpm migrateagainst a live database.migrate()once. The migrations are safe to rerun, and nobody has to remember to bump a version.pnpm migratedoesn't take the new advisory lock, so running it while a Worker is migrating could still race. Running it before the deploy, as the guide says, is safe because the marker is already there when the new Worker starts. I left the CLI alone to keep its behavior unchanged.