streaming: support exact publication on TTL streams - #86
Merged
Conversation
Keep idempotency and recovery metadata aligned with fixed and sliding stream lifetimes so retry-safe publication does not require changing a stream's retention contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A keyed stream publication now shares the same finite lifetime as the stream it enters. This lets callers use retry-safe publication on fixed- and sliding-TTL streams without changing retention or crashing on a deadline mismatch.
Observed failure
Stream.AddOncepreviously required an absolute-deadline stream. A consumer creates session streams that are bounded to 50,000 entries and expire 72 hours after the latest event. When a normal event established that sliding-TTL stream and a fresh publisher then calledAddOnce, Pulse rejected the call because the stream had no immutable absolute deadline. The caller treated the failed publication as an invariant violation and terminated.Contract after this change
AddOnceaccepts any stream generation with finite retention:AddorAddOncepublishes an event.The idempotency key still identifies the exact length-delimited event name, topic, and payload. An exact retry returns the original Redis event ID. Reusing the key with changed content still returns
ErrIdempotencyConflict. Explicit retention still has to match the active generation, and an unbounded generation is still rejected because retry identity would have no finite removal point.Pulse owns this behavior because it owns the Redis keys for stream data, generation identity, expiry, retry records, and recovery metadata. Callers continue to send the same
AddandAddOnceinputs and receive the same IDs and errors; they no longer need to replace a sliding lifetime with an absolute deadline merely to make retries safe.Compatibility and rollout
This is additive for fixed- and sliding-TTL streams. Existing absolute-deadline behavior and persisted retention strings are unchanged. No data migration or deployment ordering is required: Pulse is linked into each consuming binary, and existing Redis generations are adopted using their stored retention contract. Rolling back the library restores the prior TTL rejection without rewriting Redis state.
Validation
go test ./streaming -count=1go test ./streaming -run '^Test(AddOnceConcurrentClientsPublishExactlyOnce|AddOnceMetadataSurvivesMaxLenAndScriptFlush|AddOnceAdoptsPreGenerationFlatStreamInPlace|AddOnceSharesFixedAndSlidingTTLLifetimes|StreamTTL.*)$' -count=1staticcheck ./streaming/...The main review points are
addOnceScriptinstreaming/exact_publication.go, the sliding resource refresh instreaming/stream_lifecycle.go, and the fixed-versus-sliding counterexample inTestAddOnceSharesFixedAndSlidingTTLLifetimes.