fix(agentex): end SSE subscriptions on terminal task / vanished topic; stop deleting shared stream - #385
Draft
deepthi-rao-scale wants to merge 1 commit into
Draft
Conversation
deepthi-rao-scale
force-pushed
the
fix/agentex-sse-zombie-subscription-leak
branch
from
July 29, 2026 18:16
89b915c to
2846998
Compare
…hared stream Task event SSE subscriptions had no terminal condition. stream_task_events ran a `while True` loop whose only exits were client disconnect (CancelledError) and fatal errors. Once a task finished its producer stopped writing, but the reader kept blocking on the topic forever — issuing an XREAD every couple of seconds and pinning one connection from the shared per-process Redis pool. Because the stream key carries a sliding TTL, a finished task's key eventually disappears while readers keep blocking on it, turning each subscription into a permanent zombie. Accumulated zombies exhaust the pool, which is shared with the readiness probe, so /readyz fails while the dependency-free /healthz keeps returning 200 — the pod goes Unready and is never restarted. Termination: - Primary is event-driven. Every terminal transition funnels through task_service.transition_status (and delete via update_task), which XADDs a task_updated event carrying the new status onto the task's own stream. The reader already delivers those events, so it returns as soon as it forwards one whose task is in a terminal status — no DB lookup and no ordering race, since the terminal event is the last message produced. - Fallback runs on the keepalive-ping cadence for the cases the event path cannot see: a client that connects after the task already finished, a producer that dies without emitting a terminal event, or a task_updated with no task payload. It ends the stream when the task is terminal or when a topic that had existed is reclaimed. stream_ever_existed is seeded from the tail snapshot so a mid-flight subscriber correctly arms the vanished-topic check. Shared stream: - Drop the per-subscriber cleanup_stream in finally. The topic is keyed only by task id and shared by every viewer, so deleting it on one subscriber's exit tore the stream out from under the others. The sliding TTL already reclaims it. Also adds stream_exists to the stream port + Redis adapter (EXISTS) for the vanished-topic fallback, and two integration regression tests: the stream ends on its own once the task is terminal, and a single subscriber disconnecting does not delete the shared topic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deepthi-rao-scale
force-pushed
the
fix/agentex-sse-zombie-subscription-leak
branch
from
July 29, 2026 18:35
2846998 to
2eed3af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was broken
When you open a task in the UI, the backend keeps a live stream open to push updates. That stream had no way to end itself — it only stopped if the browser disconnected.
So when a task finished, the backend just kept listening forever, holding one Redis connection each time. These pile up until the pool is empty, at which point the pod can't serve requests (and its health check fails, so it sits stuck until a manual restart).
A second bug: when one viewer left, the code deleted the task's shared event stream — yanking it out from under everyone else watching the same task.
The fix
task_updatedevent when a task finishes, so the stream now closes as soon as it sees a terminal status.Tests
Two integration regression tests: the stream ends on its own once the task is terminal, and one viewer disconnecting no longer deletes the shared stream.
Related: the UI-side half of this leak is #383.
🤖 Generated with Claude Code