Skip to content

fix: lock dead_letter rows in dlq_replay#287

Merged
NikolayS merged 1 commit into
mainfrom
claude/fix-dlq-replay-race-oqxpbr
Jul 8, 2026
Merged

fix: lock dead_letter rows in dlq_replay#287
NikolayS merged 1 commit into
mainfrom
claude/fix-dlq-replay-race-oqxpbr

Conversation

@NikolayS

Copy link
Copy Markdown
Owner

Bug

pgque.dlq_replay(i_dead_letter_id) selected the pgque.dead_letter row (joined to pgque.queue) without for update, then called pgque.insert_event(), then deleted the row. Two concurrent dlq_replay() calls for the same dl_id (the function is granted to pgque_writer, so any two writers) could both pass the unlocked select and both call insert_event() — the dead-lettered event was re-enqueued twice. Both deletes then "succeeded" (the loser's delete removed 0 rows, silently). pgque.dlq_replay_all() had the same unlocked-select shape in its loop query.

Fix

  • dlq_replay(): the initial select now ends with for update of dl, locking only the dead_letter row (not the joined pgque.queue row). The second concurrent caller blocks on the row lock; after the first transaction commits its delete, the second's select re-evaluates under read committed, finds no row, and the existing if not found branch raises the existing error: dead letter entry not found: <id>. No behavior change for non-concurrent callers.
  • dlq_replay_all(): the loop query now uses for update of dl skip locked.

Locking-choice rationale for dlq_replay_all: skip locked fits the "replay everything" semantics better than a blocking for update. A row locked by a concurrent dlq_replay()/dlq_replay_all() is already being handled by that session; blocking would only make this call wait so it could either replay the row a second time (the exact race being fixed) or count a guaranteed not found-style failure. Skipping leaves the row to its owner; if that owner rolls back, the row is still in the DLQ for the next replay pass.

Error messages, grants, security definer + set search_path = pgque, pg_catalog, and SQL style are unchanged. Generated files sql/pgque.sql and sql/pgque-tle.sql are regenerated via bash build/transform.sh and committed together with the source change (only the dlq chunk differs).

TDD / verification

New deterministic two-session harness, following the pattern of tests/two_session_receive_lock.sh: tests/two_session_dlq_replay_race.sh. Session 1 runs begin; select pgque.dlq_replay(<dl_id>); pg_sleep(4); commit;; once session 1 is observed inside pg_sleep via pg_stat_activity, session 2 calls pgque.dlq_replay(<dl_id>) for the same id. The harness then asserts session 2 fails with dead letter entry not found, exactly 1 event of the replayed type is received, and the DLQ is empty.

Red (unfixed code, origin/main install):

$ PGQUE_TEST_DSN=postgresql:///pgque_dlq_oqxpbr tests/two_session_dlq_replay_race.sh
FAIL: session2 replay succeeded; expected 'dead letter entry not found' after waiting on session1
--- session1.out ---
 s1_new_eid=2003
--- session2.out ---
 s2_new_eid=2004

Both sessions re-enqueued the same dead letter (two new event ids).

Green (fixed build, fresh install):

$ psql -d pgque_dlq_oqxpbr -v ON_ERROR_STOP=1 -f sql/pgque.sql   # exit 0
$ PGQUE_TEST_DSN=postgresql:///pgque_dlq_oqxpbr tests/two_session_dlq_replay_race.sh
PASS: concurrent dlq_replay serialized; second caller got 'dead letter entry not found' and the event was re-enqueued exactly once

Full regression suite (fresh DB, fixed build):

$ psql -d pgque_dlq_oqxpbr -v ON_ERROR_STOP=1 -f tests/run_all.sql   # exit 0
153 PASS notices, including test_api_dlq and test_dlq_edge_cases

Generated-file sync: bash build/transform.sh after the source edit; git status showed only sql/pgque-additions/dlq.sql, sql/pgque.sql, sql/pgque-tle.sql, and the new test changed; the embedded chunks match the source edit.

Manual verification command for reviewers:

createdb pgque_dlq_check
psql -d pgque_dlq_check -v ON_ERROR_STOP=1 -f sql/pgque.sql
PGQUE_TEST_DSN=postgresql:///pgque_dlq_check tests/two_session_dlq_replay_race.sh
psql -d pgque_dlq_check -v ON_ERROR_STOP=1 -f tests/run_all.sql

Addresses finding A3 of #283

https://claude.ai/code/session_01KAaEGkQZmey1D1xCsVGmqv


Generated by Claude Code

Two concurrent pgque.dlq_replay() calls for the same dl_id could
both pass the unlocked existence select, both call insert_event(),
and re-enqueue the dead-lettered event twice; the second delete then
silently removed 0 rows. pgque.dlq_replay_all() had the same
unlocked-select shape.

dlq_replay() now locks the dead_letter row with 'for update of dl':
the second caller blocks, re-evaluates after the first commits, finds
no row, and raises the existing 'dead letter entry not found' error.
dlq_replay_all() uses 'for update of dl skip locked' so a bulk replay
skips rows already being replayed by a concurrent session instead of
blocking or double-replaying them.

Adds tests/two_session_dlq_replay_race.sh, a deterministic two-session
harness that fails on the unfixed code (event enqueued twice) and
passes with the row lock (one event, clean error for the loser).

Verification:
  bash build/transform.sh
  psql -d <db> -v ON_ERROR_STOP=1 -f sql/pgque.sql
  PGQUE_TEST_DSN=postgresql:///<db> tests/two_session_dlq_replay_race.sh
  psql -d <db> -v ON_ERROR_STOP=1 -f tests/run_all.sql

Addresses finding A3 of #283.

https://claude.ai/code/session_01KAaEGkQZmey1D1xCsVGmqv
@NikolayS

NikolayS commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

samorev Code Review Report

Pipeline Coverage
PASS Not reported

BLOCKING ISSUES (1)

HIGH MR/PR state - Review target is draft

The review target is still marked as draft.
Fix: Mark it ready for review before merge.


Summary

Area Findings Potential Filtered
CI/Pipeline 0 0 0
Security 0 0 0
Bugs 0 0 0
Tests 0 0 0
Guidelines 0 0 0
Docs 0 0 0
Metadata 1 0 0

Note:

  • Findings: High-confidence issues (8-10/10) - blocking or non-blocking per severity
  • Potential: Medium-confidence issues (4-7/10) - review manually
  • Filtered: Low-confidence issues (0-3/10) - excluded as likely false positives
Review metadata
provider=github
kind=pr
project=NikolayS/PgQue
number=287
target=github:NikolayS/PgQue#287
state=OPEN
draft=true
diff_lines=330
diff_added=225
diff_removed=3
diff_bytes=13004
comments_count=0
commits_count=1
ci_status=success
ci_summary=total=14 success=14 failure=0 pending=0 other=0
prompt=.claude/commands/review-mr.md
blocking=false
posted_by=gh
no_comment=false
live_posting=posted

samorev-assisted review (AI analysis by Tanya301/samorev)

@NikolayS NikolayS marked this pull request as ready for review July 8, 2026 21:55
@NikolayS NikolayS merged commit a4bd61c into main Jul 8, 2026
14 checks passed
@NikolayS NikolayS deleted the claude/fix-dlq-replay-race-oqxpbr branch July 8, 2026 21:55
@NikolayS

NikolayS commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

Post-merge verification (PR-lifecycle steps 1–3)

Verified on a fresh local PostgreSQL 18.3 install. Note: this PR was already merged into main (a4bd61c, after #292 cd3b71d) by the time this ran, so "red" is demonstrated against the pre-fix parent commit rather than current main.

1. CI — all 14 checks green on HEAD; mergeable.

2. Diff review — locking change only (for update of dl in dlq_replay, for update of dl skip locked in dlq_replay_all). security definer + set search_path = pgque, pg_catalog unchanged; no new subtransactions introduced (the pre-existing begin/exception in dlq_replay_all is untouched, and neither function is a hot path). bash build/transform.sh after init'ing the pgq submodule produced zero driftsql/pgque.sql and sql/pgque-tle.sql match the source edit in sql/pgque-additions/dlq.sql.

3. Real testing

Race reproduced against the pre-fix build (cd3b71d):

FAIL: session2 replay succeeded; expected 'dead letter entry not found' after waiting on session1
 s1_new_eid=2003
 s2_new_eid=2004          <- event re-enqueued twice

Same test against the merged fix build:

PASS: concurrent dlq_replay serialized; second caller got 'dead letter entry not found' and the event was re-enqueued exactly once

Full regression suite on the fix build: tests/run_all.sql=== ALL TESTS PASSED === (153 PASS, exit 0).

Fix confirmed correct.

@NikolayS

NikolayS commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

Post-merge code review — PR #287 (fix: lock dead_letter rows in dlq_replay)

Thorough post-merge review (the review bot may not have run properly on merge). Reviewed the full diff at merge commit a4bd61c, read the surrounding DLQ source, and verified the fix empirically on Postgres 17. Verdict: the fix is correct and complete. No blocking findings. Two nits below, neither warranting a follow-up on its own.

Empirical verification

Ran the new harness against both the merged build and the pre-fix parent (cd3b71d) on a fresh postgres:17:

  • Red (parent cd3b71d, new harness against old install): FAILS — both sessions re-enqueue the dead letter (s1_new_eid=2003, s2_new_eid=2004). The bug reproduces exactly as described.
  • Green (merged a4bd61c): PASSES — session 2 blocks, then raises dead letter entry not found, event re-enqueued exactly once, DLQ empty.
  • Full suite (tests/run_all.sql, fresh DB, merged build): exit 0, === ALL TESTS PASSED ===, 156 PASS.
  • Generated-file consistency: bash build/transform.sh on the merged commit produces zero diff — sql/pgque.sql, sql/pgque-tle.sql, and the sql/pgque-additions/dlq.sql source are in sync.

(a) Concurrency correctness — correct

  • for update of dl (sql/pgque-additions/dlq.sql:139) locks only the dead_letter row, not the joined pgque.queue row. This is the right scope: locking the queue row would create needless contention with queue/tick operations.
  • The READ COMMITTED re-check is sound. When session 2's for update waits on session 1's locked row and session 1 commits a delete of that row, PostgreSQL's EvalPlanQual drops the row from the result set — so into v_dl finds nothing, if not found fires, and the existing dead letter entry not found error is raised. No replay-of-deleted-row anomaly. Confirmed empirically (green).
  • Disjoint event sets still run concurrently: dlq_replay(dl_id) locks a single row selected by dl.dl_id, so two calls for different dl_ids never contend. No false serialization.
  • No deadlock / lock-ordering issue. dlq_replay only ever locks one dead_letter row, then inserts + deletes that same row; it never waits for a second dead_letter row lock, so it cannot form a cycle with dlq_purge (bulk delete) or a second dlq_replay. dlq_replay_all uses skip locked and therefore never blocks on a dead_letter row at all. event_dead() only inserts new rows (on conflict do nothing), touching different tuples than a replay of an existing row.

(b) Coverage of replay entry points — complete

The only two re-enqueue paths are dlq_replay (single) and dlq_replay_all (batch); both are fixed. dlq_inspect is read-only, dlq_purge is a bulk delete, event_dead is the insert side. No uncovered replay path exists.

(c) Test quality — genuine concurrency regression test

tests/two_session_dlq_replay_race.sh uses two real backends (backgrounded psql holding an open transaction in pg_sleep), and gates session 2 on observing session 1 inside PgSleep via pg_stat_activity — deterministic, not a fixed sleep. It fails on pre-fix code and passes on the fix (verified above). It asserts all three properties: the loser errors with the exact message, exactly one event is re-enqueued, and the DLQ ends empty. Follows the established tests/two_session_receive_lock.sh pattern.

Nits (non-blocking)

  • nit — dlq_replay_all under-counts on a concurrent rollback. sql/pgque-additions/dlq.sql:198 (for update of dl skip locked): if a row is locked by a concurrent dlq_replay() that later rolls back, this call skips it — the row is counted as neither replayed nor failed, and remains in the DLQ for a later pass. This is the correct choice for replay-everything semantics and is documented in the function's docstring, so callers relying on replayed/failed totals just see a transient under-report. No action needed; flagging only so it's on record.
  • nit — comment block style. CLAUDE.md prefers /* ... */ for comments spanning 2+ lines. The new 5–6 line header comments use -- per line. However they match the dominant convention already used throughout dlq.sql (e.g. the pre-existing event_dead / dlq_replay_all headers), so matching surrounding style arguably wins here. Cosmetic.

Reviewed with REV methodology; SOC2 findings intentionally out of scope per repo policy.

@NikolayS

NikolayS commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

samorev Code Review Report

Pipeline Coverage
PASS Not reported

This report supersedes the earlier samorev review --fetch comment on this PR. That comment was a CI/draft gate check only and performed no code analysis (it hardcoded zeros in every area). The results below come from a genuine per-area review of the diff, verified empirically with a two-session race reproduction on PostgreSQL 17.

No blocking issues. The fix is correct and complete; two non-blocking notes below.

What was verified

  • Red/green race repro: against the pre-fix parent, the new harness tests/two_session_dlq_replay_race.sh fails — both sessions re-enqueue the dead letter (s1_new_eid=2003, s2_new_eid=2004). Against the merged fix it passes — session 2 blocks, raises dead letter entry not found, and the event is re-enqueued exactly once.
  • Concurrency correctness: for update of dl locks only the dead_letter row (not the joined pgque.queue row) — correct scope. The READ COMMITTED re-check is sound: when the waiting session's for update resolves against a row session 1 deleted, EvalPlanQual drops it, if not found fires, and the existing error is raised. Disjoint dl_ids never contend; no lock-ordering cycle with dlq_purge/event_dead.
  • Coverage: the only two re-enqueue paths (dlq_replay, dlq_replay_all) are both fixed; dlq_inspect/dlq_purge/event_dead are not replay paths.
  • Full suite: tests/run_all.sql=== ALL TESTS PASSED === (156 PASS) on a fresh DB with the merged build.
  • Generated-file consistency: bash build/transform.sh at the merge commit produces zero drift; sql/pgque.sql, sql/pgque-tle.sql, and sql/pgque-additions/dlq.sql are in sync. security definer + set search_path unchanged; no new subtransactions in any hot path.

NON-BLOCKING (2)

LOW sql/pgque-additions/dlq.sql:198 - dlq_replay_all under-counts on a concurrent rollback

for update of dl skip locked skips a row locked by a concurrent dlq_replay() that later rolls back; the row is counted as neither replayed nor failed and stays in the DLQ for a later pass.
Suggestion: This is the correct choice for replay-everything semantics and is documented in the docstring — no change needed. Callers relying on the totals just see a transient under-report. Flagged for the record.

INFO sql/pgque-additions/dlq.sql - multi-line comment style

The new 5-6 line header comments use -- per line where CLAUDE.md prefers /* ... */ for 2+ lines.
Suggestion: Matches the dominant convention already in dlq.sql (event_dead/dlq_replay_all headers), so matching surrounding style is defensible. Cosmetic.


Summary

Area Findings Potential Filtered
CI/Pipeline 0 0 0
Security 0 0 0
Bugs 1 0 0
Tests 0 0 0
Guidelines 1 0 0
Docs 0 0 0
Metadata 0 0 0

Note:

  • Findings: High-confidence issues (8-10/10) - blocking or non-blocking per severity
  • Potential: Medium-confidence issues (4-7/10) - review manually
  • Filtered: Low-confidence issues (0-3/10) - excluded as likely false positives
Review metadata
provider=github
kind=pr
project=NikolayS/pgque
number=287
target=github:NikolayS/pgque#287
state=MERGED
draft=false
diff_lines=330
diff_added=225
diff_removed=3
diff_bytes=13012
comments_count=3
commits_count=1
ci_status=success
ci_summary=total=14 success=14 failure=0 pending=0 other=0
prompt=.claude/commands/review-mr.md
blocking=false
posted_by=gh
no_comment=false
live_posting=posted

samorev-assisted review (AI analysis by Tanya301/samorev)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants