[fix][broker] Prevent duplicate geo-replicated messages after target topic reload#25860
Open
void-ptr974 wants to merge 1 commit into
Open
[fix][broker] Prevent duplicate geo-replicated messages after target topic reload#25860void-ptr974 wants to merge 1 commit into
void-ptr974 wants to merge 1 commit into
Conversation
665700d to
18bde31
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #25861
Motivation
This fixes a correctness issue in persistent geo-replication V2 deduplication. The issue is in the core replication data path and can produce duplicate messages on the target cluster during normal recovery/failover paths.
Geo-replication V2 deduplication uses the source topic position as the target-side dedup watermark. For a replicated message, the source replicator adds
__MSG_PROP_REPL_SOURCE_POSITION=<source-ledger-id>:<source-entry-id>, and the target broker stores the latest replicated source position as:<replicator-producer>_LID<replicator-producer>_EIDThis state is not normal producer sequence state. It is the target-side checkpoint used to identify whether a replayed source entry has already been persisted.
There are three related problems in the current handling of this watermark:
The geo V2 watermark is not recovered from replayed target entries
When deduplication is enabled or a target topic is loaded, the broker restores dedup state from the
pulsar.dedupcursor snapshot and then replays entries after that snapshot.The existing replay path restores normal producer sequence ids from message metadata, but it does not rebuild geo V2
_LID/_EIDstate from replicated messages. If the target topic is unloaded after replicated messages are persisted but before the latest dedup snapshot includes their source positions, the in-memory geo watermark is lost.If the source replicator later reconnects and replays from its replication cursor, the target broker can fail to identify those source entries as duplicates and persist them again.
The geo V2 watermark is stored as two separate snapshot keys
A geo V2 watermark is only valid when
_LIDand_EIDare restored together. Saving only one side does not represent a usable source position.The snapshot logic previously treated all dedup entries as independent producer states. This makes the geo watermark vulnerable to partial persistence or being crowded out by normal producer sequence state. After reload, an incomplete
_LID/_EIDpair cannot safely deduplicate source replay.The geo V2 watermark can be removed by normal producer inactivity cleanup
Normal producer dedup state can be purged after producer inactivity. Geo V2
_LID/_EIDstate has a different lifecycle: it is a source-position watermark for replication, not the lifecycle state of the replicator producer.The source replicator may disconnect, reconnect, or fail over and replay the same source entries from its replication cursor. If the target has purged the geo watermark because the replicator producer was inactive, those replayed source entries can be accepted and written again.
A concrete failure window is:
This does not require client misuse or corrupted input. Source cursor replay is an expected recovery behavior, so target-side geo dedup state must survive topic reload and producer inactivity cleanup.
Modifications
replicatedFromand__MSG_PROP_REPL_SOURCE_POSITIONfrom persisted replicated messages._LID/_EIDpairs in dedup snapshots.Verifying this change
This change added tests and can be verified as follows:
Result:
BUILD SUCCESSFUL.Does this pull request potentially affect one of the following parts: