Skip to content

Keep the receive-handler consumer alive until ServeHTTP returns - #1313

Open
imu2008 wants to merge 1 commit into
cloudevents:mainfrom
imu2008:fix/1224-consumer-pairing
Open

Keep the receive-handler consumer alive until ServeHTTP returns#1313
imu2008 wants to merge 1 commit into
cloudevents:mainfrom
imu2008:fix/1224-consumer-pairing

Conversation

@imu2008

@imu2008 imu2008 commented Jul 31, 2026

Copy link
Copy Markdown

Fixes #1224.

The invariant that was broken

Protocol.ServeHTTP hands the inbound message over on the unbuffered Protocol.incoming
channel and blocks there until a consumer picks it up. NewHTTPReceiveHandler starts exactly
one consumer goroutine per ServeHTTP call, so the two have to stay paired.

That consumer waited on the request context. When a client disconnects during the
hand-off, the consumer exits while the message it was meant to pick up is still pending, and
the pairing is permanently offset by one: from then on every request is served by the next
request's consumer, so each one blocks until further traffic arrives.

That is also why it is easy to miss — on a busy endpoint the next request arrives within
milliseconds and nothing looks wrong; on a quiet one requests hang until the client or the
gateway gives up. Restarting the process is the only way to clear it.

The fix

Wait on a context that is cancelled when ServeHTTP returns, instead of the request context.
While a message is pending its producer has not returned yet, so at least one consumer is
guaranteed to still be waiting for it.

Why the cancellation cannot simply be dropped

This looked like the obvious fix at first and it is wrong: Protocol.ServeHTTP has paths that
return without sending anything —

  • rate-limiter rejection (!ok → 429)
  • http.MethodOptions
  • http.MethodGet

Protocol.incoming is never closed, and net/http cancels the request context when the
handler returns, so today that cancellation is what lets the consumer of such a request exit.
An unconditionally waiting consumer would block forever on every OPTIONS request, GET request
and rate-limited request.

Cancelling on ServeHTTP return covers both cases: it releases consumers whose request never
produced a message, and it keeps alive those whose message is still in flight.

Scope

Two lines of behaviour change in v2/client/http_receiver.go; the rest of the diff is the
comment explaining the invariant. No change to protocol/http, no new exported API, and
nothing that requires a newer Go version than the module already declares.

Invoke still receives req.Context(), so the context seen by receiver functions is
unchanged. The new context is only ever passed to Respond, which uses it solely for the
select — it reads no values and propagates no deadline — and its lifetime is bounded by the
handler via defer, so a consumer can never outlive its ServeHTTP call. Deriving it from
context.Background() rather than from the request context keeps this working on the Go
version the module declares (context.WithoutCancel is Go 1.21).

Test

TestEventReceiverServeHTTP_ConsumerPairing in v2/client/http_receiver_test.go.

It serves a request whose context is already cancelled, then two healthy ones, and asserts
each ServeHTTP returns and each event reaches the receiver function. Without the fix the
first ServeHTTP never returns — its own consumer is gone and no further request follows to
take the message — so the test fails on a timeout:

--- FAIL: TestEventReceiverServeHTTP_ConsumerPairing (10.00s)
    http_receiver_test.go:192: ServeHTTP did not return for request "cancelled"

With the fix it passes immediately. The full v2 suite passes; gofmt and go vet are clean.

I deliberately did not add a goroutine-count assertion for the no-send paths — it tends to be
flaky in shared test binaries. The reasoning is captured in the comment instead, and the
existing TestEventReceiverServeHTTP_Options / _Webhook tests cover those paths
functionally.

Protocol.ServeHTTP hands the inbound message over on the unbuffered
Protocol.incoming channel and blocks there until a consumer picks it up.
NewHTTPReceiveHandler starts exactly one consumer goroutine per ServeHTTP
call, so the two have to stay paired.

That consumer waited on the request context. When a client disconnects
during the hand-off the consumer exits while the message it was meant to
pick up is still pending, and the pairing is permanently offset by one:
from then on every request is served by the *next* request's consumer, so
each one blocks until further traffic arrives. On a low-traffic endpoint
this surfaces as requests hanging until the client or gateway times out,
while high traffic hides it completely. Restarting the process is the only
way to clear it.

Wait on a context that is cancelled when ServeHTTP returns instead. While
a message is pending its producer has not returned yet, so at least one
consumer is guaranteed to still be waiting for it.

The cancellation cannot simply be dropped: Protocol.ServeHTTP has paths
that return without sending anything (rate limiting, OPTIONS, GET) and
Protocol.incoming is never closed, so an unconditionally waiting consumer
would block forever on every such request. Cancelling on return covers
both cases.

Fixes cloudevents#1224

Co-developed-by: Claude <noreply@anthropic.com>
Signed-off-by: winger <panhuawenmail@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrency bug in http.Protocol when used in conjuction with client.NewHTTPReceiveHandler

1 participant