Skip to content

Add an opt-in setting that lets the API Worker apply its own schema - #59

Open
MendyLanda wants to merge 2 commits into
boringcomputers:mainfrom
MendyLanda:self-host-auto-migrate
Open

MendyLanda wants to merge 2 commits into
boringcomputers:mainfrom
MendyLanda:self-host-auto-migrate

Conversation

@MendyLanda

Copy link
Copy Markdown

What changed

Self-hosters and one-click installers such as Appflare deploy new versions of the API Worker without a checkout or a direct database URL, so they can't run pnpm migrate before each update. This adds an opt-in Worker var, MAIL_AUTO_MIGRATE_ENABLED, that lets the Worker bring its own schema up to date. It is off by default, so the hosted service keeps applying reviewed SQL by hand and behaves exactly as it does today.

Proposed in #58.

  • migrate() now records one marker row, schema-<hash>, in mail.schema_migrations. The hash covers the migration list, so any edit to a migration changes the marker. pnpm migrate and the native migration script call migrate(), so they record the same row. The schema itself does not change.
  • ensureSchema() in database.ts looks up the marker with one query. A fresh database has no mail.schema_migrations table yet, and it treats that error, 42P01, as out of date. When the marker is missing, it opens one transaction, takes pg_advisory_xact_lock on a fixed key, checks the marker again, and runs migrate(). A failure rolls the whole transaction back. Every statement runs inside that one transaction on one connection, with no session settings or session locks, so it works with Hyperdrive's transaction pooling. It never runs set local role postgres, which only works with PlanetScale's migration logins.
  • worker.ts keeps one ensureSchema() promise per isolate and awaits it before the fetch, email, queue, and scheduled handlers touch the database. A rejection clears it so the next event tries again. The event that starts the migration also passes it to waitUntil, so the migration finishes even if that event ends first and later events never wait on a dead request.
  • When the check fails, no handler runs. Fetch returns the same 500 internal_error with transient: true that a database error gets. Email throws "Email could not be stored", as for a storage failure. Queue retries the whole batch after 60 seconds, like the existing per-message database retries. Scheduled throws, so the next cron run tries again.
  • /healthz skips the check. The database guide says it does not query the database, and I don't think an uptime probe should start or wait on a migration. The cron runs every minute, so a deploy with no traffic still migrates within a minute.
  • Zod validates the var with the rest of the configuration. Anything other than true or false makes the Worker report that it is not configured.
  • The Worker guide, the root README's self-host steps, and the database guide describe the option, the rights the Hyperdrive user then needs, and that the hosted flow stays manual.

How I tested it

I ran everything with Node 24.19.0 and pnpm 10.15.1. pnpm test:postgres ran against a disposable postgres:18-bookworm container on 127.0.0.1:5435, as in the database guide.

  • pnpm build
  • pnpm check
  • pnpm test
  • pnpm source:check
  • pnpm test:postgres (when SQL changed)
$ pnpm build
apps/email build: Done            (every package printed Done)
$ pnpm check
apps/email check: Done
17 operations; OpenAPI, client types, and CLI/MCP schemas agree.
$ pnpm test
apps/email test:  Test Files  28 passed | 1 skipped (29)
apps/email test:       Tests  268 passed | 1 skipped (269)
(the other six packages: 107 tests, 0 failures)
$ pnpm source:check
40 files match Bezalel 27740c1c62a2531d865e04f1f98cb71aa83b6024; 25 declared modifications
$ TEST_DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5435/bezalel_email_test pnpm test:postgres
 Test Files  30 passed (30)
      Tests  275 passed (275)

test/auto-migrate.test.ts runs on PGlite in pnpm test and on PostgreSQL in pnpm test:postgres. It calls the Worker's real handlers and maps the Hyperdrive connection string to a test database. It checks that:

  • with the var unset or false, a request never touches mail.schema_migrations, and an invalid value reports not_configured;
  • with true, /healthz makes no query, and an empty database ends up with the same tables, indexes, functions, and markers as migrate() before the first request returns 200;
  • once the schema is current, each isolate makes exactly one marker lookup and opens no transaction;
  • four concurrent ensureSchema() calls on an empty database return one migrated and three current, and six concurrent requests in one isolate share one transaction;
  • while the database is unreachable, fetch, email, queue, and scheduled each fail without running their handler, each one retries the check, and the next request succeeds once the database is back;
  • a migration that fails partway leaves no mail objects behind, and the next call finishes it.

test/postgres-runtime.test.ts gains a workerd run through Wrangler's local Hyperdrive binding. The Hyperdrive user owns the database but can't become postgres, as on Neon or Supabase. Eight concurrent cold requests against an empty database all return 200, the mail objects belong to that user, and one schema marker is recorded. The existing pnpm migrate test now also checks that ensureSchema() returns current afterward, so the CLI and the Worker agree on the marker.

I also removed pieces of the change to see the tests catch it. With only the advisory lock removed, the concurrent test fails on PostgreSQL 18 in 3 of 3 runs with duplicate key value violates unique constraint "pg_namespace_nspname_index". Removing the recheck under the lock, the code that clears a failed promise, or the gate itself also fails the matching tests on PGlite.

Test doubles

The new tests make no provider calls. The existing doubles still apply. Cloudflare sending and routing return canned responses, the PGlite tests use a stub R2 binding and a fake queue batch, and the workerd test uses Wrangler's local R2. Wrangler's local Hyperdrive binding doesn't pool connections, so these tests don't prove behavior under deployed Hyperdrive. That's why the migration only uses transaction-scoped features.

Risks

  • Off by default. The hosted deployment, the native Worker, and the wrangler configs don't change.
  • No schema change. migrate() writes one extra row to mail.schema_migrations.
  • With the var on, an upgrade's DDL runs on the first event after the deploy and holds its locks until it commits. alter table ... add column if not exists locks the table even when the column exists, so requests can wait briefly. That's the same cost as running pnpm migrate against a live database.
  • The marker is a hash of the migration text, so any edit to a migration, even whitespace, makes the next isolate run the full migrate() once. The migrations are safe to rerun, and nobody has to remember to bump a version.
  • pnpm migrate doesn't take the new advisory lock, so running it while a Worker is migrating could still race. Running it before the deploy, as the guide says, is safe because the marker is already there when the new Worker starts. I left the CLI alone to keep its behavior unchanged.

Self-hosters and one-click installers such as Appflare deploy new
versions without a checkout or a direct database URL, so they can't run
`pnpm migrate` before each update.

With MAIL_AUTO_MIGRATE_ENABLED=true, each isolate checks a schema marker
in mail.schema_migrations once, before the first event that uses the
database. If the marker is missing, it runs migrate() through Hyperdrive
in one transaction under pg_advisory_xact_lock, without changing role.
A failure rolls back and fails the event, and the next event retries.
/healthz skips the check.

migrate() records the marker, a hash of the migration list, so
`pnpm migrate` and the Worker agree. The setting is off by default, so
the hosted deployment keeps applying reviewed SQL by hand.
@greptile-apps

greptile-apps Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Critical risk] Adds automatic schema migration through the Worker itself.

The changes since the previous review address the earlier documentation concern. No identified issues block merging.

Summary

Adds opt-in automatic schema updates through MAIL_AUTO_MIGRATE_ENABLED. The Worker checks the schema before database-backed events and retries after a failed migration.

  • Automatic migration remains off by default, preserving the hosted service's manual migration flow.
  • MendyLanda acknowledged the incomplete upgrade-permissions guidance and fixed both guides. They now explain that the Hyperdrive user must own existing mail objects, directly or through their owning role. Objects created by pnpm migrate belong to postgres, so existing deployments should keep automatic migration off unless ownership is transferred first.
  • The earlier documentation finding is fully addressed. No new issues were found in the changes since the previous review.

Reviews (2) · Last reviewed commit: "Note that automatic upgrades need the Hy..."

Comment thread apps/email/README.md Resolved
@MendyLanda

Copy link
Copy Markdown
Author

@michaelshimeles this small addition will allow this app to be installed via appflare

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant