fix(event/stream): reject a client timeout that cannot outlast PullTimeout - #11
Merged
cedricve merged 2 commits intoAug 5, 2026
Conversation
PullMessages is a long-poll: the camera holds the connection open for up to PullTimeout waiting for an event. http.Client.Timeout bounds the whole exchange — dial, write, wait-for-headers — and starts before the camera has parsed the request, so a client ceiling equal to or below PullTimeout expires first on every interval with no event. The failure mode is quiet and easy to misread. Pulls fail continuously, but the stream stays alive because ReconnectAfterFailures recreates the subscription, and each recreate makes the camera replay its full property state. Events keep arriving, in bursts, on the reconnect cadence rather than when they happen — so it reads as a slow camera rather than a misconfiguration. Observed in the field with both values at 5s: every pull timed out, recovery landed after exactly 3 failures, and ~90 property-state events were replayed every 18s. Validated in NewStream, before the subscription call, since the config can only fail. A zero client timeout stays legal — unbounded is safe because the pull loop is already bounded by ctx.
Two gaps in the previous commit's guard. Strict inequality was not enough. A client timeout one millisecond above PullTimeout passed, and the test pinned that as valid — but the client ceiling also has to cover dial, TLS and the response transfer on top of the poll it outlasts, which on a cellular bearer is hundreds of milliseconds. Require minClientHeadroom (5s) above PullTimeout. The error was a bare fmt.Errorf, so callers could not tell it from the transient pull/renew/recreate failures they retry. A consumer that retries this one loops forever on a configuration that can never succeed. ErrInvalidOptions is a sentinel they can short-circuit on. Zero stays accepted: it is the SDK's default when a caller passes no client, so rejecting it would break every default consumer. The comment no longer claims that is safe — the caller interface documents that ctx cannot interrupt an in-flight SOAP call, so an unbounded client is the one case nothing can unwedge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reject a client timeout that cannot outlast PullTimeout
Summary
PullMessagesis a long-poll: the client asks the camera to hold the connection open for up toPullTimeoutand answer early if an event arrives.http.Client.Timeoutis not symmetrical with it — it bounds the entire exchange, dial and TLS and request write and the wait for response headers, and it starts before the camera has even parsed the request. So a client ceiling equal to or belowPullTimeoutexpires before the camera's hold does, on every interval with no event. Not intermittently: deterministically.Nothing in the package noticed.
NewStreamaccepted the configuration and returned a workingStream, and the caller had no way to learn that its polls could never succeed.Why it matters
The failure is quiet rather than loud, which is what makes it worth guarding against rather than documenting. Because
ReconnectAfterFailuresrecreates the subscription after three consecutive pull failures, and a freshCreatePullPointSubscriptionmakes the camera replay its whole property state, events keep arriving — so the stream looks alive.Observed on a deployed camera with both values at 5s, over a ten-minute window:
Ordinary polling never succeeded once. Every event reaching the consumer arrived on the reconnect cadence, roughly 18 seconds apart, in bursts of replayed subscription state rather than when the underlying thing actually happened. Roughly 540 of the 546 events in that window were the same property dump repeated seven times. The pull failures are logged at debug and the recreates succeed, so at default log levels this presents as a slow camera rather than a misconfiguration, and motion-triggered consumers see latency of up to a full reconnect cycle with no error anywhere.
Changes
NewStreamnow validates the device's client ceiling against the resolvedPullTimeoutbefore issuing the subscription call, and returns an error if it cannot outlast it.Strict inequality is not sufficient. The client ceiling also has to cover dial, TLS and the response transfer on top of the poll it outlasts, which on a constrained link runs to hundreds of milliseconds — a ceiling one millisecond above
PullTimeoutloses the same race. The check therefore requires a real headroom floor,minClientHeadroom, currently 5s.The error wraps a new
ErrInvalidOptionssentinel. The existingErrPullFailed/ErrRenewFailed/ErrRecreateFailedare transient and consumers retry them; this one never becomes valid by retrying, so a consumer that cannot distinguish it will loop forever on a configuration that can never work. The sentinel lets them short-circuit.A zero client timeout is still accepted. It is what
onvif.NewDevicefills in when a caller passes no client, so rejecting it would break every default consumer including this repository's ownexamples/event/stream. The comment no longer claims that case is safe, though — thecallerinterface documents that ctx cannot interrupt an in-flight SOAP call, so an unbounded client is precisely the case nothing can unwedge.Tests
TestValidateClientTimeoutcovers the boundary in both directions: unbounded, comfortable headroom, exactly the minimum headroom, a hair under it, strictly-greater-but-no-headroom, equal values (the original bug), and a ceiling below the poll.TestErrInvalidOptions_IsDistinctFromStreamErrorspins that the sentinel does not match any of the three transient stream error types, which is the property consumers will branch on.Compatibility
This is a behavioural change at construction: a configuration that previously produced a limping stream now fails fast. That is deliberate — the configuration can only ever fail — but it will surface for any existing consumer that sets a short client timeout without having noticed the consequence, so it warrants at least a minor version.
Options.PullTimeoutand the package doc comment are updated to state the constraint.Relationship to #10
These two changes share a root cause —
PullTimeoutis the clock every other timeout in the package has to respect, and nothing enforced that — but they are independent defects with different owners and different symptoms. This one concernshttp.Client.Timeout, which the caller supplies, and breaks steady-state event delivery. #10 concerns the close-drain bound, which the package owns, and breaks shutdown by skippingUnsubscribeand orphaning the pull-point. Either can occur without the other: before this fix a consumer passing no client at all had an unbounded ceiling, so it had #10's bug and not this one.They do overlap mechanically. Both extend
NewStream, and both need to read the device's HTTP client timeout, so each branch carries its own small accessor for it. Whichever merges second will need a trivial rebase: delete the duplicate accessor and hoist the single read so it feeds both the validation here and the drain derivation there. Happy to do that rebase in whichever order suits you, or to fold the two into one PR if you would rather review them together.