feat(replication): subscription health + app_versions count canary - #2822
Conversation
Slot lag alone missed dead Google apply workers. Check subscription health on the replica and compare app_versions counts (cached 5m). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
📝 WalkthroughWalkthroughThe replication endpoint now checks subscription-worker health and ChangesReplication health monitoring
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested labels: Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant AdminDashboard
participant replicationEndpoint
participant PrimaryDatabase
participant ReadReplica
AdminDashboard->>replicationEndpoint: Request replication status
replicationEndpoint->>PrimaryDatabase: Query subscription workers and app_versions count
replicationEndpoint->>ReadReplica: Validate source and query app_versions count
replicationEndpoint-->>AdminDashboard: Return slot, subscription, and canary results
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
Comment |
Visual diff passedVisual changesGenerated at 2026-08-02T12:28:57.254Z. Threshold: 0.1% pixel difference.
Commit: Open |
Merging this PR will not alter performance
Comparing 
Footnotes
|
Typed Hono context only exposes requestId via c.get(); use the X-Database-Source header set by getPgClient instead. Co-authored-by: Cursor <cursoragent@cursor.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_ae0b7b3c-f209-4fa3-83ec-9e45c0638a0a) |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/pages/admin/dashboard/replication.vue`:
- Around line 363-367: Update the status badge classes in the replication
dashboard template to use the configured DaisyUI `d-` prefix: change the static
`badge` class and both conditional `badge-success`/`badge-error` values in the
status span.
In `@supabase/functions/_backend/public/replication.ts`:
- Around line 534-535: Update the admin request flow around
querySubscriptionHealth and getCachedDataCanary to start both independent
operations concurrently and await their combined results, preserving the
existing subscription and dataCanary assignments and behavior.
In `@tests/replication-data-canary.unit.test.ts`:
- Around line 57-74: Move the vi.useRealTimers() cleanup from the test body into
the existing afterEach hook so it runs even when assertions in the “marks
enabled subscription without apply worker as ko” test fail; keep fake-timer
setup and system-time configuration within the test.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 312dd825-4f0d-4184-93a6-b6789b7703a4
📒 Files selected for processing (4)
docs/pr-screenshots/replication-data-canary.webpsrc/pages/admin/dashboard/replication.vuesupabase/functions/_backend/public/replication.tstests/replication-data-canary.unit.test.ts
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Cap-go/capacitor-updater(manual)
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
4 issues found across 4 files
Confidence score: 2/5
- In
supabase/functions/_backend/public/replication.ts, the subscription aggregation can mark/replicationhealthy when one enabled subscription is unhealthy, which risks masking a real apply-worker outage and delaying incident response — treat every enabled subscription as required to be healthy (ignore only disabled rows). - In
supabase/functions/_backend/public/replication.ts, awaiting the Cache API write without a timeout can stall the endpoint response even after canary checks complete, creating avoidable availability risk on health probes — make cache writes best-effort with the existing timeout guard. - In
tests/replication-data-canary.unit.test.ts, unnecessary fake timers can leak on assertion failure and make test outcomes harder to trust, and insupabase/functions/_backend/public/replication.tssequential independent checks add latency to uncached requests — remove fake timers in this test and run the independent checks withPromise.allto de-risk behavior and response time.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/replication-data-canary.unit.test.ts">
<violation number="1" location="tests/replication-data-canary.unit.test.ts:58">
P3: The fake timers here are unnecessary — evaluateSubscriptionHealth is called with the default checkedAt and the assertions only inspect status/reasons. Because vi.useRealTimers() runs only after the assertion, a failed assertion leaves fake timers active for subsequent tests in this file (afterEach only clears the canary cache and never restores timers). Drop vi.useFakeTimers()/vi.setSystemTime()/vi.useRealTimers(), or restore timers in afterEach for isolation.</violation>
</file>
<file name="supabase/functions/_backend/public/replication.ts">
<violation number="1" location="supabase/functions/_backend/public/replication.ts:152">
P1: An unhealthy enabled subscription is hidden whenever any sibling is healthy, so `/replication` can report green while an apply worker is down. Disabled rows can be ignored, but every enabled subscription should be healthy for the aggregate check to pass.</violation>
<violation number="2" location="supabase/functions/_backend/public/replication.ts:409">
P2: A Cache API stall can keep `/replication` from responding even after the canary counts succeed because the cache write is awaited without a timeout. The cache write is best-effort here, so bounding it with the existing timeout keeps caching from becoming a health-check latency dependency.</violation>
<violation number="3" location="supabase/functions/_backend/public/replication.ts:534">
P3: The subscription health check and data canary check are independent but awaited sequentially, adding the subscription query latency to every uncached canary request. Consider running them with `Promise.all` to reduce latency on the admin request path.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Require every enabled subscription to be healthy, split worker/receipt signals, parallelize replica checks, bound cache writes, and fix DaisyUI badge classes. Co-authored-by: Cursor <cursoragent@cursor.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_db789043-67cf-4923-a635-e37d75eed8b6) |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/pages/admin/dashboard/replication.vue`:
- Line 367: Update the subscription status cell in the replication dashboard to
check sub.enabled before health status: render a neutral DISABLED badge for
disabled subscriptions, while preserving the existing success/error rendering
for enabled subscriptions.
In `@supabase/functions/_backend/public/replication.ts`:
- Around line 546-551: Update the response construction around
querySubscriptionHealth and getCachedDataCanary so their failures do not alter
the existing top-level status or HTTP status code, which must remain based on
slot health. Expose subscription and canary failure state through a separate
aggregate field, or gate any response-shape change behind an explicit version
mechanism, while preserving existing /replication clients’ behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 3b9addf6-d93b-4521-b7be-ddc6a93399a6
📒 Files selected for processing (3)
src/pages/admin/dashboard/replication.vuesupabase/functions/_backend/public/replication.tstests/replication-data-canary.unit.test.ts
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Cap-go/capacitor-updater(manual)
There was a problem hiding this comment.
All reported issues were addressed across 4 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Mark disabled subscription rows as disabled (not KO), fail empty replica counts, and expose slot_status while keeping overall health inclusive of subscription and canary checks. Co-authored-by: Cursor <cursoragent@cursor.com>
Review follow-up (AI generated)Addressed remaining review threads:
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit bd25412. Configure here.
| checked_at: checkedAt, | ||
| threshold_seconds: thresholdSeconds, | ||
| subscriptions, | ||
| reasons: reasons.length > 0 ? reasons : ['subscription_unhealthy'], |
There was a problem hiding this comment.
Healthy subscription lists unhealthy reason
Medium Severity
When every enabled subscription passes, evaluateSubscriptionHealth still sets top-level reasons to subscription_unhealthy because an empty reasons array triggers the fallback. The admin dashboard subtitle prefers reasons over the healthy subscription name, so a green Subscription card can show a failure message.
Reviewed by Cursor Bugbot for commit bd25412. Configure here.
| const replicaSource = c.res.headers.get('X-Database-Source') ?? '' | ||
| if (!isReplicaDatabaseSource(replicaSource)) { | ||
| subscription = skippedSubscription('no_replica_connection') | ||
| dataCanary = skippedDataCanary('no_replica_connection') |
There was a problem hiding this comment.
Replica detection ignores context source
Medium Severity
After opening the read client, replica checks gate on X-Database-Source from the response headers only. Elsewhere getPgClient records the chosen binding in databaseSource via c.set, and header mutation is best-effort. A stale or missing header after the primary client ran first can mark checks as no_replica_connection even when the read pool target is a replica.
Reviewed by Cursor Bugbot for commit bd25412. Configure here.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@supabase/functions/_backend/public/replication.ts`:
- Around line 580-585: Preserve the existing top-level status meaning and HTTP
status behavior by deriving both from slotStatus in the replication response.
Keep the aggregate subscription/canary result in a new overall_status field,
update consumers such as the dashboard to read overall_status for aggregate
health, and retain slot_status without changing existing response compatibility.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: ce9f1fa3-5f92-4404-93f9-2bb61c45d411
📒 Files selected for processing (3)
src/pages/admin/dashboard/replication.vuesupabase/functions/_backend/public/replication.tstests/replication-data-canary.unit.test.ts
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Cap-go/capacitor-updater(manual)
| // Intentionally includes subscription + canary: /replication is the admin health probe. | ||
| const overallStatus: SlotStatus = failingChecks.some(Boolean) ? 'ko' : 'ok' | ||
|
|
||
| const response = { | ||
| status: overallStatus, | ||
| slot_status: slotStatus, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Preserve the existing meaning of top-level status.
Line 584 changes status from slot health to aggregate health. Existing /replication clients can now interpret a healthy slot state as failed. Adding slot_status does not preserve clients that only read status.
Keep status and the HTTP status code based on slotStatus. Add a new aggregate field, such as overall_status, for subscription and canary failures. Update the dashboard to use that new field.
As per coding guidelines: “Public API and plugin changes must remain backward compatible: do not remove or change existing fields, meanings, formats, or status codes; use plugin version detection when behavior must differ.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@supabase/functions/_backend/public/replication.ts` around lines 580 - 585,
Preserve the existing top-level status meaning and HTTP status behavior by
deriving both from slotStatus in the replication response. Keep the aggregate
subscription/canary result in a new overall_status field, update consumers such
as the dashboard to read overall_status for aggregate health, and retain
slot_status without changing existing response compatibility.
Source: Coding guidelines
There was a problem hiding this comment.
Risk: medium. Left a non-blocking comment — Cursor Bugbot finished as skipped and reported 2 unresolved findings on subscription health / replica detection, and this replication infra change is above the low auto-approve threshold. Assigned reviewers for human review.
Sent by Cursor Approval Agent: Pull Request Approver External
|
1 participant
Summary (AI generated)
/replicationto check Google SQL subscription apply workers (enabled + livepid+ receipt lag), not only primary slot lagapp_versionsrow counts between primary and read replica, fail when replica is empty or drift exceeds 1%COUNT(*)runs at most once per 5 minutesMotivation (AI generated)
Primary slot lag can stay green while the subscriber apply worker is dead or pointed at the wrong subscription name. That left OTA updates reading stale replica data with no alert from
/replication.Business Impact (AI generated)
Faster detection of dead/stale read replicas reduces production OTA outages and support load when replication silently stops applying.
Test Plan (AI generated)
bunx vitest run tests/replication-data-canary.unit.test.ts/replicationas platform admin with replica bindings: expectsubscription.status=okanddata_canarywith primary/replica counts/replicationtwice within 5 minutes: second response should showdata_canary.cached=true503andstatus=ko/admin/dashboard/replicationshows Subscription + Data canary cardsGenerated with AI
Made with Cursor
Summary by CodeRabbit
New Features
Documentation
Tests