Partition keys (1A) + producer idempotency (1B): spec, brief, draft implementation

[NikolayS]

Status: DRAFT — do not merge

Awaiting external review by Fabrizio (Supabase Storage) — this implements the two features he needs to move off pg-boss. Merge only after his review + the standard loop (CI → REV → functional test).

What's here

Design layer (converged through 3 internal review rounds + Max + Fabrizio + two stronger-model refinement passes):

• blueprints/partition-keys/SPEC.md v0.7 — ordered per-key consumption via N slot subscriptions; G1/G2 corrected (session claim is load-bearing, not "pure liveness"); mechanism/policy seam; member/heartbeat table rejected (D10); tick-gated online-resize protocol as sanctioned future path (D9, draft); connection-pooler caveat (only the claim connection needs session mode).
• blueprints/idempotency/SPEC.md — producer TTL dedup as a separate send-layer feature.
• Brief deployed at https://pgque.dev/briefs/partition-keys.html with full version history.

User stories — US-12.1–12.7 (lifecycle events: keyed send, per-key order, cross-key parallelism, single processor, claim/crash-recovery, observability, enforced N) and US-13.1–13.5 (migrations: TTL dedup, effect-scoped keys, expiry, GC, mutual-exclusion recipe) in both SPECs, the brief, and executable acceptance tests.

Draft implementation (red/green TDD, engine untouched):

• sql/pgque-api/partition_keys.sql — 4-arg send, subscribe_slot/unsubscribe_slot (persisted enforced N), receive_partitioned (SECURITY DEFINER co-ownership over the admin-only get_batch_cursor extra_where hook), ack_partitioned, slot_lock_key/claim_slot/release_slot, partition_slot_status view.
• sql/pgque-api/send_idem.sql — atomic single-statement TTL claim, dedup returns the original event_id, maint_idem() GC self-registered via queue_extra_maint (zero maint.sql edits).
• Wired into build/transform.sh; sql/pgque.sql + pgque-tle.sql rebuilt.

Reproduction & benchmarks (earlier commits): TS/bun repro of both workloads incl. the version-suppression hazard guardrail; bloat-under-backlog + throughput bench vs a pg-boss-style mutable job table.

Verification (fresh PostgreSQL 18.3)

1. install (\i sql/pgque.sql)                      OK
2. tests/run_all.sql                               ALL TESTS PASSED (173 PASS)
3. tests/acceptance/run_acceptance.sql (US-1..13)  ALL ACCEPTANCE TESTS PASSED
4. tests/two_session_slot_claim.sh                 PASS (claim exclusivity across
   real sessions; dead session's slot instantly reclaimable; owner_pid visible)

Repro commands:

psql -d pgque_test -f sql/pgque.sql
psql -d pgque_test -f tests/run_all.sql
psql -d pgque_test -f tests/acceptance/run_acceptance.sql
PGQUE_TEST_DSN=postgresql://...:5499/pgque_test tests/two_session_slot_claim.sh

Known follow-ups before leaving draft

• T-security non-superuser-installer variant (co-ownership currently exercised as superuser only)
• CI across PG 14–17 (locally verified on 18 only)
• Online resize (D9) is spec-only, marked draft-pending-its-own-review — deliberately NOT implemented
• pause policy (Phase 2) blocked on O1 — deliberately NOT implemented

🤖 Generated with Claude Code

https://claude.ai/code/session_01HYTw9GXVXJvectNqZ2fqsq

[NikolayS]

Partition-keys benchmark — v0.8 lease model at the target scale (SPEC R2/R7, S4)

Question this answers: does the N-slot + batch-lease design sustain a Supabase-Storage-like profile — >400M events/day, ordered per tenant — and what do read amplification (R2) and rotation pinning (R7) cost in practice?

Setup

• Hardware: Hetzner CCX43 — 16 dedicated vCPU (AMD), 64 GiB RAM, local NVMe, Ubuntu 24.04.
• PostgreSQL 18 (PGDG): shared_buffers=16GB, synchronous_commit=off, wal_level=minimal, max_wal_size=16GB, checkpoint_timeout=15min, wal_compression=on, jit=off, pg_stat_statements preloaded.
• pgque: this PR's sql/pgque.sql at v0.8 (batch-lease slot ownership).
• Workload: pgbench producer at rate-controlled 5,000 ev/s (= 432M events/day), ~200-byte JSON payloads, partition key tenant-<n> drawn random_zipfian(1, 2000, 1.1) — hot-head tenant skew. Consumers: one bun worker per slot driving the real API — claim_slot (30 s TTL) → sticky receive_partitioned (batch 500) → ack_partitioned — i.e. the documented lease loop, no shortcuts. Ticker every 250 ms, maint() every 60 s, queue rotation period 30 s.
• Harness: benchmark/partition-keys/ in this PR (samplers: per-slot lag 5 s, sys 10 s, pg_stat_statements 10 s, per-table bloat 30 s).

Headline

Phase	N	Producer ev/s	Consume ev/s	Pending p99 (events)	Buffers/event	CPU peak	Peak dead tup (meta)	Event tbl GiB end
steady-16 (30 min)	16	5,000	4,999	1,320	4.39	26%	7,586	3.57
steady-32 (30 min)	32	5,001	5,000	1,318	7.02	31%	9,502	7.19
stalled-16 (15 min, slot 7 SIGSTOP 8 min)	16	4,997	4,996	1,665,413	2.97	30%	8,912	5.38

Steady state: the design holds the target with a wide margin. ~9.0M events produced and consumed per 30-minute phase; per-slot pending p50 0, p99 ~1,320 events (≈ 0.26 s of production); zero buffer reads (fully cache-resident window); CPU peak 26–31% on 16 cores. Doubling N from 16→32 at the same rate changed steady-state lag not at all.

Read amplification (R2)

N	buffers/event	32/16 ratio
16	4.39	—
32	7.02	1.60× (ideal-linear would be 2×)

Every slot scans the full stream and filters server-side, so buffers touched per produced event grow with N — observed sublinear (1.60× for 2× slots), all from shared_buffers at this scale. Practical read: at 5k ev/s, going from 16 to 32 slots costs ~5 CPU points. R2 is real, measurable, and affordable at this profile; it argues for keeping N at the parallelism you need, as the SPEC says.

Rotation pinning (R7) — demonstrated, quantified

Slot 7's worker SIGSTOPped for 8 minutes mid-phase:

• Its lag grew at 2,663 events/s (its zipfian share of the stream), peaking at 2.39M pending.
• Rotation floor pinned: event tables stopped dropping; event-table bytes grew 3.62 → 5.38 GiB during the stall and resumed dropping after.
• Other 15 slots: completely unaffected (p50 pending stayed 0).
• Catch-up did not complete within the remaining 5 minutes after resume — a stalled slot's backlog drains at (slot consume rate − slot arrival rate), so recovery is not instant at this volume.

Two operational findings worth pinning in the docs:

1. The freed slot was not taken over. The stopped worker's lease expired after its 30 s TTL, but with M=N sticky workers each bound to its own slot, nobody polled slot 7 — it sat ownerless for ~15 min. This is exactly SPEC §15's "every slot must be polled" requirement showing its teeth: client claim loops should cycle idle/free slots (or run M>N candidates), not only stick to their own.
2. Stale subscriptions are not inert (found by accident): a 4-slot consumer left over from a smoke run silently pinned rotation for the whole queue while every live metric looked healthy. The per-slot lag alert on partition_slot_status is mandatory, and unsubscribe_slot on teardown is not optional. (The harness now force-drops the bench queue at setup for this reason.)

Dead tuples & bloat

• Event tables: 0 dead tuples throughout all phases. Append-only + rotation-by-drop holds at 432M/day equivalent — the core pgque thesis.
• partition_slot (the v0.8 lease, 2 UPDATEs/batch): peak 557–704 dead tuples, steady ~96 KiB across all phases. HOT keeps lease churn negligible — the "no per-interval heartbeat churn" claim (D10/D11) is confirmed empirically.
• subscription / tick metadata churn peaked at ~9.5k / ~7.2k dead tuples and returned to ~0 — bounded rotation-metadata behavior.
• Total DB footprint stayed = live event window (3.6–7.2 GiB, proportional to rotation retention), no growth beyond it except during the deliberate R7 stall.

Caveats

• Handler is trivial (count + ack): this measures the queue's capacity, not handler capacity; real deployments add their own per-event work.
• Producer and consumers ran on the DB host over localhost — no network latency in the loop.
• 30-minute steady phases; no multi-day soak here (the 2-hour held-xmin soak methodology exists separately in benchmark/).
• Catch-up-after-stall was measured but not to completion; drain-time modeling for a lagged slot is worth a follow-up note.

Bottom line for the >400M/day question: steady state sustains it at ~26–31% peak CPU on 16 dedicated cores with sub-second per-slot lag and zero event-table bloat, at both N=16 and N=32. The two things that need operational discipline are exactly the two the SPEC already names: keep every slot polled (R7 + §15) and alert on per-slot lag.

Raw data: /tmp/bench/pk/ CSVs (per-slot lag 5 s, pgss 10 s, bloat 30 s, sys 10 s) — archived before VM teardown; harness committed in benchmark/partition-keys/.