Skip to content

fix(event/stream): reject a client timeout that cannot outlast PullTimeout - #11

Merged
cedricve merged 2 commits into
kerberos-io:masterfrom
sharedjourney:fix/guard-pull-timeout-vs-client-timeout
Aug 5, 2026
Merged

fix(event/stream): reject a client timeout that cannot outlast PullTimeout#11
cedricve merged 2 commits into
kerberos-io:masterfrom
sharedjourney:fix/guard-pull-timeout-vs-client-timeout

Conversation

@ttradesman

Copy link
Copy Markdown

Reject a client timeout that cannot outlast PullTimeout

Summary

PullMessages is a long-poll: the client asks the camera to hold the connection open for up to PullTimeout and answer early if an event arrives. http.Client.Timeout is not symmetrical with it — it bounds the entire exchange, dial and TLS and request write and the wait for response headers, and it starts before the camera has even parsed the request. So a client ceiling equal to or below PullTimeout expires before the camera's hold does, on every interval with no event. Not intermittently: deterministically.

Nothing in the package noticed. NewStream accepted the configuration and returned a working Stream, and the caller had no way to learn that its polls could never succeed.

Why it matters

The failure is quiet rather than loud, which is what makes it worth guarding against rather than documenting. Because ReconnectAfterFailures recreates the subscription after three consecutive pull failures, and a fresh CreatePullPointSubscription makes the camera replay its whole property state, events keep arriving — so the stream looks alive.

Observed on a deployed camera with both values at 5s, over a ten-minute window:

11:39:25 fail  11:39:31 fail  11:39:37 fail → recovered (~90 events)
11:39:43 fail  11:39:49 fail  11:39:55 fail → recovered (~90 events)
11:40:00 fail  11:40:06 fail  11:40:12 fail → recovered (~90 events)

Ordinary polling never succeeded once. Every event reaching the consumer arrived on the reconnect cadence, roughly 18 seconds apart, in bursts of replayed subscription state rather than when the underlying thing actually happened. Roughly 540 of the 546 events in that window were the same property dump repeated seven times. The pull failures are logged at debug and the recreates succeed, so at default log levels this presents as a slow camera rather than a misconfiguration, and motion-triggered consumers see latency of up to a full reconnect cycle with no error anywhere.

Changes

NewStream now validates the device's client ceiling against the resolved PullTimeout before issuing the subscription call, and returns an error if it cannot outlast it.

Strict inequality is not sufficient. The client ceiling also has to cover dial, TLS and the response transfer on top of the poll it outlasts, which on a constrained link runs to hundreds of milliseconds — a ceiling one millisecond above PullTimeout loses the same race. The check therefore requires a real headroom floor, minClientHeadroom, currently 5s.

The error wraps a new ErrInvalidOptions sentinel. The existing ErrPullFailed / ErrRenewFailed / ErrRecreateFailed are transient and consumers retry them; this one never becomes valid by retrying, so a consumer that cannot distinguish it will loop forever on a configuration that can never work. The sentinel lets them short-circuit.

A zero client timeout is still accepted. It is what onvif.NewDevice fills in when a caller passes no client, so rejecting it would break every default consumer including this repository's own examples/event/stream. The comment no longer claims that case is safe, though — the caller interface documents that ctx cannot interrupt an in-flight SOAP call, so an unbounded client is precisely the case nothing can unwedge.

Tests

TestValidateClientTimeout covers the boundary in both directions: unbounded, comfortable headroom, exactly the minimum headroom, a hair under it, strictly-greater-but-no-headroom, equal values (the original bug), and a ceiling below the poll. TestErrInvalidOptions_IsDistinctFromStreamErrors pins that the sentinel does not match any of the three transient stream error types, which is the property consumers will branch on.

Compatibility

This is a behavioural change at construction: a configuration that previously produced a limping stream now fails fast. That is deliberate — the configuration can only ever fail — but it will surface for any existing consumer that sets a short client timeout without having noticed the consequence, so it warrants at least a minor version.

Options.PullTimeout and the package doc comment are updated to state the constraint.

Relationship to #10

These two changes share a root cause — PullTimeout is the clock every other timeout in the package has to respect, and nothing enforced that — but they are independent defects with different owners and different symptoms. This one concerns http.Client.Timeout, which the caller supplies, and breaks steady-state event delivery. #10 concerns the close-drain bound, which the package owns, and breaks shutdown by skipping Unsubscribe and orphaning the pull-point. Either can occur without the other: before this fix a consumer passing no client at all had an unbounded ceiling, so it had #10's bug and not this one.

They do overlap mechanically. Both extend NewStream, and both need to read the device's HTTP client timeout, so each branch carries its own small accessor for it. Whichever merges second will need a trivial rebase: delete the duplicate accessor and hoist the single read so it feeds both the validation here and the drain derivation there. Happy to do that rebase in whichever order suits you, or to fold the two into one PR if you would rather review them together.

PullMessages is a long-poll: the camera holds the connection open for
up to PullTimeout waiting for an event. http.Client.Timeout bounds the
whole exchange — dial, write, wait-for-headers — and starts before the
camera has parsed the request, so a client ceiling equal to or below
PullTimeout expires first on every interval with no event.

The failure mode is quiet and easy to misread. Pulls fail continuously,
but the stream stays alive because ReconnectAfterFailures recreates the
subscription, and each recreate makes the camera replay its full
property state. Events keep arriving, in bursts, on the reconnect
cadence rather than when they happen — so it reads as a slow camera
rather than a misconfiguration.

Observed in the field with both values at 5s: every pull timed out,
recovery landed after exactly 3 failures, and ~90 property-state events
were replayed every 18s.

Validated in NewStream, before the subscription call, since the config
can only fail. A zero client timeout stays legal — unbounded is safe
because the pull loop is already bounded by ctx.
Two gaps in the previous commit's guard.

Strict inequality was not enough. A client timeout one millisecond
above PullTimeout passed, and the test pinned that as valid — but the
client ceiling also has to cover dial, TLS and the response transfer on
top of the poll it outlasts, which on a cellular bearer is hundreds of
milliseconds. Require minClientHeadroom (5s) above PullTimeout.

The error was a bare fmt.Errorf, so callers could not tell it from the
transient pull/renew/recreate failures they retry. A consumer that
retries this one loops forever on a configuration that can never
succeed. ErrInvalidOptions is a sentinel they can short-circuit on.

Zero stays accepted: it is the SDK's default when a caller passes no
client, so rejecting it would break every default consumer. The comment
no longer claims that is safe — the caller interface documents that ctx
cannot interrupt an in-flight SOAP call, so an unbounded client is the
one case nothing can unwedge.
@ttradesman ttradesman changed the title fix(event/stream): guard pull timeout vs client timeout fix(event/stream): reject a client timeout that cannot outlast PullTimeout Jul 23, 2026
@cedricve
cedricve merged commit 3af23e5 into kerberos-io:master Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants