Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,11 @@ lists the API Worker's settings. In the dashboard's file:
2. Create the database and connect Hyperdrive with the
[database guide](docs/planetscale.md). Put a direct connection string in
`apps/email/.env` as `DATABASE_URL` and run `pnpm migrate`. It applies the
full schema and is safe to rerun on upgrades.
full schema and is safe to rerun on upgrades. To skip this step on install
and on every upgrade, set `MAIL_AUTO_MIGRATE_ENABLED` to `true` in the API
Worker's `vars`, and the Worker applies the schema itself. It is off by
default, and the Hyperdrive user then needs the right to create the schema.
The [Worker guide](apps/email/README.md#set-up-cloudflare) has the details.
3. Pick a dashboard. The Cloudflare Worker dashboard supports public
passwordless accounts; follow the [account guide](docs/accounts.md). The
Node dashboard below uses one shared owner password and can see every inbox,
Expand Down
22 changes: 18 additions & 4 deletions apps/email/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,15 +61,29 @@ flowchart LR
Create a dedicated PlanetScale Postgres database, connect its primary to
Hyperdrive with query caching disabled, and replace the `HYPERDRIVE` ID in
`wrangler.jsonc`. Run migrations with a direct PostgreSQL `DATABASE_URL`
from `apps/email/.env`. The Worker uses the binding and needs no database URL secret.
from `apps/email/.env`, or let the Worker apply them as described below.
The Worker uses the binding and needs no database URL secret.
9. Deploy with `pnpm --filter @bezalel/email run deploy`. In Cloudflare Email
Routing, send the domain's catch-all to this Worker. Remove any specific
address rules that would override this route for agent addresses.

Run `migrate` before deploying an upgrade too. Applied schema changes are recorded
in `mail.schema_migrations`. The search-index upgrade replaces the old generated
column and its index in one transaction, preserves messages, and can be rerun.
It takes a table lock while rebuilding the index; schedule it during a quiet period.
in `mail.schema_migrations`, with a marker for the current schema. The
search-index upgrade replaces the old generated column and its index in one
transaction, preserves messages, and can be rerun. It takes a table lock while
rebuilding the index; schedule it during a quiet period.

To skip the manual step, set `MAIL_AUTO_MIGRATE_ENABLED` to `true` in `vars`.
It is off by default. Each Worker isolate then checks the marker once, before
the first request, email, queue batch, or cron run that uses the database. If
the marker is missing, the isolate runs the migrations through Hyperdrive in one
transaction under an advisory lock, so isolates that start together migrate
once. A failed migration rolls back and fails the event, and the next event
tries again. `/healthz` skips the check. An upgrade's migrations run on the
first event after the deploy. The Hyperdrive user needs the right to create the
`mail` schema and its objects, for example as the database owner. The Worker
never switches to the `postgres` role the way `pnpm migrate` does. The hosted
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Resolved
service leaves this off and applies reviewed SQL by hand.

The Worker checks sending and receiving before creating an inbox. The default
mode requires a catch-all targeting this Worker. For a subdomain sharing a zone
Expand Down
33 changes: 33 additions & 0 deletions apps/email/src/database.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { createHash } from "node:crypto"
import { Client } from "pg"
import { migrations } from "./schema.js"

Expand Down Expand Up @@ -38,6 +39,38 @@ export const postgresDatabase = (connectionString: string): Database => ({
}),
})

// Any edit to the migrations changes this marker, so a database that has it is up to date.
const schemaVersion = `schema-${createHash("sha256").update(JSON.stringify(migrations)).digest("hex").slice(0, 16)}`

export const migrate = async (db: DatabaseQueries): Promise<void> => {
for (const statement of migrations) await db.query(statement)
await db.query("insert into mail.schema_migrations(id) values ($1) on conflict do nothing", [schemaVersion])
}

const schemaRecorded = async (db: DatabaseQueries): Promise<boolean> => {
const [row] = await db.query<{ recorded: boolean }>(
"select exists(select 1 from mail.schema_migrations where id = $1) as recorded", [schemaVersion])
return row?.recorded === true
}

// Brings the schema up to date through the Worker's own connection. A current database costs one
// query. Otherwise one transaction holds a fixed advisory lock, so isolates that start together
// migrate once, and a failure rolls back everything. It runs as the connecting user and never
// changes role, unlike `pnpm migrate`.
export const ensureSchema = async (db: Database): Promise<"current" | "migrated"> => {
try {
if (await schemaRecorded(db)) return "current"
} catch (error) {
// 42P01: a fresh database has no mail.schema_migrations table yet.
if ((error as { code?: unknown }).code !== "42P01") throw error
}
return db.transaction(async tx => {
await tx.query("select pg_advisory_xact_lock(hashtext('bezalel-email-schema'))")
// Another isolate may have finished while this one waited for the lock.
const [table] = await tx.query<{ present: boolean }>(
"select to_regclass('mail.schema_migrations') is not null as present")
if (table?.present && await schemaRecorded(tx)) return "current"
await migrate(tx)
return "migrated"
})
}
35 changes: 32 additions & 3 deletions apps/email/src/worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import {
type MailConfig,
type Operation
} from "./contracts.js"
import { postgresDatabase } from "./database.js"
import { ensureSchema, postgresDatabase, type Database } from "./database.js"
import { MailService } from "./mail-service.js"
import { MailboxStore } from "./mailbox-store.js"
import { CustomDomains, type GatewayConfig } from "./custom-domains.js"
Expand Down Expand Up @@ -55,6 +55,7 @@ export interface Env {
MAIL_DOMAIN_ENCRYPTION_KEY?: string
MAIL_INBOUND_SCAN_ENABLED?: string
MAIL_FEEDBACK_SIGNERS?: string
MAIL_AUTO_MIGRATE_ENABLED?: string
AUTH_PUBLIC_URL?: string
AUTH_SECRET?: string
AUTH_PROXY_SECRET?: string
Expand All @@ -75,7 +76,8 @@ const configuration = z.object({
PUBLIC_EMAIL_URL: z.url(),
MAIL_EVENTS_URL: z.url().optional(),
BEZALEL_EVENTS_URL: z.url().optional(),
WORKER_NAME: z.string().min(1)
WORKER_NAME: z.string().min(1),
MAIL_AUTO_MIGRATE_ENABLED: z.enum(["true", "false"]).optional()
})

export const serviceFor = (env: Env): MailService => {
Expand Down Expand Up @@ -151,6 +153,23 @@ export const serviceFor = (env: Env): MailService => {
})
}

let schema: Promise<unknown> | undefined

// With MAIL_AUTO_MIGRATE_ENABLED=true, each isolate brings the schema up to date once, before the
// first event that uses the database. A failure clears the promise so the next event retries.
const schemaReady = (env: Env, db: Database, ctx?: ExecutionContext): Promise<unknown> => {
if (env.MAIL_AUTO_MIGRATE_ENABLED !== "true") return Promise.resolve()
if (!schema) {
schema = ensureSchema(db).catch((error: unknown) => {
schema = undefined
throw error
})
// Finish the migration even if the event that started it ends first.
ctx?.waitUntil(schema.catch(() => {}))
}
return schema
}

const json = (data: unknown, status = 200): Response =>
Response.json(data, { status, headers: { "cache-control": "no-store" } })

Expand Down Expand Up @@ -296,10 +315,15 @@ export async function consumeDeliveryBatch(
}

export default {
async fetch(request: Request, env: Env): Promise<Response> {
async fetch(request: Request, env: Env, ctx?: ExecutionContext): Promise<Response> {
try {
const service = serviceFor(env)
const path = new URL(request.url).pathname
// Health checks stay off the database.
if (path !== "/healthz") {
try { await schemaReady(env, service.store.db, ctx) }
catch { return json({ error: { message: "Email operation failed", code: "internal_error", transient: true } }, 500) }
}
if (path.startsWith("/api/auth/") || path.startsWith("/account-rpc/")) {
const config = accountConfig(env)
const accounts = postgresAccountAuth(config, env.HYPERDRIVE.connectionString, service)
Expand Down Expand Up @@ -327,6 +351,7 @@ export default {
): Promise<void> {
const service = serviceFor(env)
try {
await schemaReady(env, service.store.db, ctx)
const raw = await readBytes(message.raw, 25 * 1024 * 1024)
await service.acceptIncoming(message.to, raw, message.from)
ctx.waitUntil(service.processIncoming().then(() => service.processTriage()))
Expand All @@ -342,6 +367,9 @@ export default {
},
async queue(batch: MessageBatch<unknown>, env: Env, ctx: ExecutionContext): Promise<void> {
const service = serviceFor(env)
// Like a database error, a schema that is not ready retries the batch after a minute.
try { await schemaReady(env, service.store.db, ctx) }
catch { batch.retryAll({ delaySeconds: 60 }); return }
await consumeDeliveryBatch(batch, service.store, service.config)
ctx.waitUntil(service.flushEvents())
},
Expand All @@ -351,6 +379,7 @@ export default {
ctx: ExecutionContext
): Promise<void> {
const service = serviceFor(env)
await schemaReady(env, service.store.db, ctx)
ctx.waitUntil(service.processIncoming().then(() => service.flushEvents()))
ctx.waitUntil(service.processTriage())
if (env.AUTH_PUBLIC_URL && env.AUTH_FROM) ctx.waitUntil(sendNotifications(service.store.db, service.transport, env.AUTH_FROM, env.AUTH_PUBLIC_URL))
Expand Down
172 changes: 172 additions & 0 deletions apps/email/test/auto-migrate.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
import { randomUUID } from "node:crypto"
import { afterEach, describe, expect, it, vi } from "vitest"
import { ensureSchema, migrate, type Database, type DatabaseQueries } from "../src/database.js"
import type { Env } from "../src/worker.js"
import { testDatabase } from "./database.js"
import { config } from "./support.js"

// The Worker reaches each test database through its Hyperdrive connection string.
const hyperdrive = vi.hoisted(() => new Map<string, Database>())
vi.mock("../src/database.js", async (original) => {
const actual = await original<typeof import("../src/database.js")>()
return { ...actual, postgresDatabase: (url: string) => hyperdrive.get(url) ?? actual.postgresDatabase(url) }
})

const closers: Array<() => Promise<void>> = []
afterEach(async () => {
for (const close of closers.splice(0)) await close()
})

// Records every statement and can fail the ones a test picks.
const database = async () => {
const { db, pg } = await testDatabase()
closers.push(() => pg.close())
const statements: string[] = []
const state = { transactions: 0, fail: undefined as ((text: string) => boolean) | undefined }
const queries = (target: DatabaseQueries): DatabaseQueries => ({
async query<T>(text: string, parameters?: unknown[]) {
statements.push(text)
if (state.fail?.(text)) throw new Error("database unavailable")
return target.query<T>(text, parameters)
},
})
const tracked: Database = {
...queries(db),
transaction: task => {
state.transactions++
return db.transaction(tx => task(queries(tx)))
},
}
const url = `postgres://hyperdrive.test/${randomUUID()}`
hyperdrive.set(url, tracked)
return { db: tracked, statements, state, url }
}

// A fresh module graph stands in for a new isolate with nothing memoized.
const isolate = async () => {
vi.resetModules()
return (await import("../src/worker.js")).default
}

const environment = (url: string, setting?: string): Env => ({
HYPERDRIVE: { connectionString: url } as Hyperdrive,
MAIL_API_TOKEN: config.apiToken, MAIL_WEBHOOK_SECRET: config.webhookSecret,
CLOUDFLARE_API_TOKEN: "test-token", CLOUDFLARE_ACCOUNT_ID: "a".repeat(32),
EMAIL_DOMAINS: JSON.stringify(config.domains), DEFAULT_EMAIL_DOMAIN: config.defaultDomain,
PUBLIC_EMAIL_URL: config.publicUrl, WORKER_NAME: "auto-migrate-test", MAIL_OBJECTS: {} as R2Bucket,
MAIL_AUTO_MIGRATE_ENABLED: setting,
})
const context = () => ({ waitUntil: vi.fn(), passThroughOnException: vi.fn() }) as unknown as ExecutionContext
const listInboxes = (authorized = true) => new Request("https://mail.example.com/rpc/listInboxes", {
method: "POST", headers: authorized ? { authorization: `Bearer ${config.apiToken}` } : {}, body: "{}",
})
const schemaObjects = (db: DatabaseQueries) => db.query(`
select c.relname as name, c.relkind::text as kind from pg_class c
join pg_namespace n on n.oid = c.relnamespace where n.nspname = 'mail'
union all select p.proname, 'function' from pg_proc p
join pg_namespace n on n.oid = p.pronamespace where n.nspname = 'mail'
order by 1, 2`)
const markers = (db: DatabaseQueries) => db.query("select id from mail.schema_migrations order by id")
const markerLookup = "select exists(select 1 from mail.schema_migrations where id = $1) as recorded"

describe("MAIL_AUTO_MIGRATE_ENABLED", () => {
it("leaves the database alone unless it is true", async () => {
for (const setting of [undefined, "false"]) {
const { db, statements, state, url } = await database()
const worker = await isolate()
expect((await worker.fetch(listInboxes(), environment(url, setting), context())).status).toBe(500)
expect(statements.some((text) => text.includes("schema_migrations"))).toBe(false)
expect(state.transactions).toBe(0)
expect(await db.query("select to_regclass('mail.schema_migrations') as marker")).toEqual([{ marker: null }])
}
const { url } = await database()
const response = await (await isolate()).fetch(listInboxes(), environment(url, "yes"), context())
expect(response.status).toBe(503)
expect(await response.json()).toMatchObject({ error: { code: "not_configured" } })
})

it("applies the full schema to an empty database before it serves the first request", async () => {
const { db, statements, url } = await database()
const reference = await testDatabase()
closers.push(() => reference.pg.close())
await migrate(reference.db)
const worker = await isolate()
const env = environment(url, "true")
expect((await worker.fetch(new Request("https://mail.example.com/healthz"), env, context())).status).toBe(200)
expect(statements).toEqual([])
const ctx = context()
const response = await worker.fetch(listInboxes(), env, ctx)
expect(response.status).toBe(200)
expect(await response.json()).toMatchObject({ result: { inboxes: [] } })
expect(ctx.waitUntil).toHaveBeenCalledTimes(1)
expect(await schemaObjects(db)).toEqual(await schemaObjects(reference.db))
expect(await markers(db)).toEqual(await markers(reference.db))
expect(statements.some((text) => /set\s+(local\s+)?role/i.test(text))).toBe(false)
})

it("costs one marker lookup per isolate once the schema is current", async () => {
const { db, statements, state, url } = await database()
await migrate(db)
statements.length = 0
const env = environment(url, "true")
const first = await isolate()
expect((await first.fetch(listInboxes(false), env, context())).status).toBe(401)
expect((await first.fetch(listInboxes(false), env, context())).status).toBe(401)
const second = await isolate()
expect((await second.fetch(listInboxes(false), env, context())).status).toBe(401)
expect(statements).toEqual([markerLookup, markerLookup])
expect(state.transactions).toBe(0)
expect(await ensureSchema(db)).toBe("current")
})

it("migrates an empty database once when isolates and requests start together", async () => {
const isolates = await database()
const results = await Promise.all(Array.from({ length: 4 }, () => ensureSchema(isolates.db)))
expect(results.sort()).toEqual(["current", "current", "current", "migrated"])
expect(await markers(isolates.db)).toHaveLength(2)

const { state, url } = await database()
const worker = await isolate()
const env = environment(url, "true")
const responses = await Promise.all(Array.from({ length: 6 }, () => worker.fetch(listInboxes(), env, context())))
expect(responses.map((response) => response.status)).toEqual(Array(6).fill(200))
expect(state.transactions).toBe(1)
})

it("fails every event while the schema cannot be checked and retries on the next one", async () => {
const { statements, state, url } = await database()
state.fail = () => true
const worker = await isolate()
const env = environment(url, "true")
const response = await worker.fetch(listInboxes(), env, context())
expect(response.status).toBe(500)
expect(await response.json()).toEqual({ error: {
message: "Email operation failed", code: "internal_error", transient: true,
} })
const setReject = vi.fn()
await expect(worker.email({
from: "customer@example.net", to: "agent@example.com", raw: new Response("mail").body!, rawSize: 4, setReject,
} as unknown as ForwardableEmailMessage, env, context())).rejects.toThrow("Email could not be stored")
expect(setReject).not.toHaveBeenCalled()
const message = { body: {}, ack: vi.fn(), retry: vi.fn() }
const batch = { messages: [message], retryAll: vi.fn(), ackAll: vi.fn() }
await worker.queue(batch as unknown as MessageBatch<unknown>, env, context())
expect(batch.retryAll).toHaveBeenCalledWith({ delaySeconds: 60 })
expect(message.ack).not.toHaveBeenCalled()
expect(message.retry).not.toHaveBeenCalled()
await expect(worker.scheduled({} as ScheduledController, env, context())).rejects.toThrow("database unavailable")
expect(statements).toEqual(Array(4).fill(markerLookup))
state.fail = undefined
expect((await worker.fetch(listInboxes(), env, context())).status).toBe(200)
})

it("rolls back a migration that fails partway and finishes it on the next attempt", async () => {
const { db, state } = await database()
state.fail = (text) => text.includes("create table if not exists mail.sends")
await expect(ensureSchema(db)).rejects.toThrow("database unavailable")
expect(await db.query("select to_regclass('mail.inboxes') as inboxes")).toEqual([{ inboxes: null }])
state.fail = undefined
expect(await ensureSchema(db)).toBe("migrated")
expect(await ensureSchema(db)).toBe("current")
})
})
2 changes: 1 addition & 1 deletion apps/email/test/fixtures/postgres-worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,6 @@ export default {
await worker.scheduled({} as ScheduledController, env, ctx)
return Response.json({ scheduled: true })
}
return worker.fetch(request, env)
return worker.fetch(request, env, ctx)
}
} satisfies ExportedHandler<Env>
6 changes: 5 additions & 1 deletion apps/email/test/native-migration.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,11 @@ it("upgrades the deployed native schema without changing saved mail, credentials

await migrate(f.db)
await migrate(f.db)
expect(await snapshot(f.db, columns)).toEqual(before)
const after = await snapshot(f.db, columns)
// The only new row is the schema marker that migrate() records.
expect(after.schema_migrations).toEqual([...before.schema_migrations!,
{ row: { id: expect.stringMatching(/^schema-[a-f0-9]{16}$/), applied_at: expect.any(String) } }])
expect({ ...after, schema_migrations: before.schema_migrations }).toEqual(before)
expect(await f.db.query("select * from mail.customer_inboxes")).toEqual([])
expect(await f.db.query("select * from mail.messages where triage is not null")).toEqual([])
expect((await f.service.execute("listInboxes", {}) as any).inboxes).toHaveLength(2)
Expand Down
Loading