Skip to content

streaming: support exact publication on TTL streams - #86

Merged
raphael merged 1 commit into
mainfrom
fix/add-once-sliding-ttl
Aug 25, 2026
Merged

streaming: support exact publication on TTL streams#86
raphael merged 1 commit into
mainfrom
fix/add-once-sliding-ttl

Conversation

@raphael

@raphael raphael commented Aug 25, 2026

Copy link
Copy Markdown
Member

A keyed stream publication now shares the same finite lifetime as the stream it enters. This lets callers use retry-safe publication on fixed- and sliding-TTL streams without changing retention or crashing on a deadline mismatch.

Observed failure

Stream.AddOnce previously required an absolute-deadline stream. A consumer creates session streams that are bounded to 50,000 entries and expire 72 hours after the latest event. When a normal event established that sliding-TTL stream and a fresh publisher then called AddOnce, Pulse rejected the call because the stream had no immutable absolute deadline. The caller treated the failed publication as an invariant violation and terminated.

Contract after this change

AddOnce accepts any stream generation with finite retention:

  • An absolute-deadline generation keeps its existing deadline. Ordinary and keyed publications never extend it.
  • A fixed-TTL generation gives the event, idempotency record, and recovery metadata the stream's remaining lifetime. Later publications do not extend them.
  • A sliding-TTL generation refreshes the stream, every idempotency record, and recovery metadata together whenever either Add or AddOnce publishes an event.

The idempotency key still identifies the exact length-delimited event name, topic, and payload. An exact retry returns the original Redis event ID. Reusing the key with changed content still returns ErrIdempotencyConflict. Explicit retention still has to match the active generation, and an unbounded generation is still rejected because retry identity would have no finite removal point.

Pulse owns this behavior because it owns the Redis keys for stream data, generation identity, expiry, retry records, and recovery metadata. Callers continue to send the same Add and AddOnce inputs and receive the same IDs and errors; they no longer need to replace a sliding lifetime with an absolute deadline merely to make retries safe.

Compatibility and rollout

This is additive for fixed- and sliding-TTL streams. Existing absolute-deadline behavior and persisted retention strings are unchanged. No data migration or deployment ordering is required: Pulse is linked into each consuming binary, and existing Redis generations are adopted using their stored retention contract. Rolling back the library restores the prior TTL rejection without rewriting Redis state.

Validation

  • go test ./streaming -count=1
  • go test ./streaming -run '^Test(AddOnceConcurrentClientsPublishExactlyOnce|AddOnceMetadataSurvivesMaxLenAndScriptFlush|AddOnceAdoptsPreGenerationFlatStreamInPlace|AddOnceSharesFixedAndSlidingTTLLifetimes|StreamTTL.*)$' -count=1
  • staticcheck ./streaming/...

The main review points are addOnceScript in streaming/exact_publication.go, the sliding resource refresh in streaming/stream_lifecycle.go, and the fixed-versus-sliding counterexample in TestAddOnceSharesFixedAndSlidingTTLLifetimes.

Keep idempotency and recovery metadata aligned with fixed and sliding stream lifetimes so retry-safe publication does not require changing a stream's retention contract.
@raphael
raphael merged commit aea9ab0 into main Aug 25, 2026
5 of 6 checks passed
@raphael
raphael deleted the fix/add-once-sliding-ttl branch August 25, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant