Drain WebSocket connections instead of dropping them

[coderloganli]

connect holds the long-lived connections and could not leave the cluster without damage. Run() blocked in ListenAndServe and never returned, so the signal.Notify in main.go was unreachable for that module and SIGTERM took Go's default path.

Three pieces of state outlived the process:

• the sockets, torn down by the kernel, so no client could tell a planned restart from a network failure;
• the etcd registration, whose TTL is two minutes, during which task kept routing to an instance that no longer existed and dropping the messages — the @todo at task/push.go:38;
• the Redis routing map.

Being able to add a replica is not worth much if removing one costs a two-minute hole.

What changes

On a signal, connect deregisters from etcd and refuses new connections first, then sends every live connection a 1001 Going Away close frame, then lets the existing disconnect path clear Redis — bounded by a five second cap split into per-step budgets. The order is the substance of it: closing connections while task is still routing here, or while new clients are still arriving, does not converge.

The buckets could not drive this. Bucket.Put runs only after authentication succeeds, so a connection that has been accepted but not authenticated belongs to no bucket — and during a restart those are exactly the connections present. connect/registry.go tracks every connection from accept to teardown, sharded the way the buckets are.

Every module now starts its work and returns a stopper rather than blocking or handling signals itself, so whether a module reacts to SIGTERM stops depending on whether its Run happened to have returned. That is also what fixes the tracer being shut down at startup: as long as Run() returned while the process kept running, any defer in it fired at the wrong time.

Three pre-existing bugs this had to fix

• DisConnect never deleted the userId -> serverId key. Not just on shutdown — an ordinary disconnect left it for its TTL to collect. It now deletes it, in a Lua script that compares first, so a departing instance cannot delete the mapping of a user who has already reconnected elsewhere. The script's three outcomes also guard the room bookkeeping, which was removing users who were online on another instance.
• Bucket.DeleteChannel deleted by user rather than by connection, so with two connections for one user the older one's teardown removed the newer one's entry, leaving that user connected but undeliverable. A drain provokes it, because every client reconnects while the old connection is still being torn down.
• connect/server_tcp.go dialled etcd from an init(), which made the whole package impossible to unit test.

Measured

make drain-demo runs both arms on one image with one environment variable between them, the method ADR 0009 used. Evidence in loadtest/reports/drain/.

	control	graceful
Closed with 1001 going away	0 (0%)	25 (100%)
Closed with another close code	25 (100%, i.e. 1006)	0
Time to notice, p50 / p95	46.7 ms / 48.7 ms	52.1 ms / 52.1 ms
etcd registration outlived the stop by	still present after 20 s	683 ms
Room messages sent	99	99
Frames received on the surviving replica	2,475	3,100

Time to notice is unchanged, and that is the honest reading. The TCP close arrives at the same moment either way. This makes a client able to tell what happened; it does not shorten the outage, and a measurement claiming otherwise is a measurement error.

The frame counts differ for a reason that checks out exactly. 99 × 25 = 2,475 is the control arm to the frame; the graceful arm's surplus is 625 = 25 × 25, each disconnect republishing room membership to the survivors. In the control arm no disconnect runs, so the remaining clients are never told the room emptied.

What this does not do

• It does not drain queued messages — the cap covers the close handshake only.
• It does not close the redelivery gap. task/push.go's @todo stays open; this narrows the window from two minutes to the length of a shutdown.
• It does not make clients reconnect. site/ is a prebuilt bundle with no source here.

Tests

37 cases written before the code: 26 unit and 12 integration, including three control-arm cases that fail if GOCHAT_GRACEFUL_SHUTDOWN ever stops switching the behaviour off.

docs/architecture.md's Known gaps is also corrected — two entries were fixed in 26f9184 and never removed, and the tracer entry named the wrong modules.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TH3DQ9ADPUsWd5Z3Cn9ckp