Skip to content

fix(retry): bound every server-controlled retry delay - #4

Merged
mzner merged 3 commits into
mainfrom
fix/cap-retry-after
Aug 9, 2026
Merged

fix(retry): bound every server-controlled retry delay#4
mzner merged 3 commits into
mainfrom
fix/cap-retry-after

Conversation

@mzner

@mzner mzner commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Problem

A retried response could dictate how long the CLI paused.

internal/httpapi/client.go and internal/webdav/client.go each parsed Retry-After and slept for exactly that duration with no ceiling. Retry-After: 86400 suspended a single retry for 24 hours.

--timeout does not recover from this. It is the per-request http.Client.Timeout, not a deadline for the whole operation, and no request is in flight while the retry loop waits. A hostile or misconfigured server could hang any command indefinitely, with nothing on screen to distinguish it from a crash. The exponential backoff had the same gap: a large --retries with a long base wait grew without bound.

Confirmed by measurement — the capped path now waits 30s where the previous code would have slept 86400s.

Why it was in two places

The policy existed as three functions (retryableStatus, retryAfter, waitRetry) duplicated verbatim across internal/httpapi and internal/webdav. A ceiling added to one copy would silently have left the other unbounded.

Fix

  • New internal/retry package owns the policy: which statuses may be retried, and how long to wait.
  • MaxDelay = 30s bounds every wait, from Retry-After or from backoff.
  • An excessive hint is clamped rather than discarded, so throttling is still respected — the retry just happens no later than the ceiling.
  • Both transports and the TUS upload path now call the shared implementation.

Retry-After parsing

Parsing now follows RFC 9110's two defined forms. The previous implementation appended "s" and called time.ParseDuration, which:

  • accepted Go duration syntax the spec does not define (5m3 became 5m3s), and
  • rejected values outright that it arguably should not (1h).

Delta-seconds now parses as an integer. An out-of-range value saturates rather than failing, so an absurd hint is capped instead of being read as "no delay requested" — failing toward a bounded retry, not toward none.

Tests

internal/retry covers the ceiling, both header forms, unusable values, whitespace, and backoff growth (96% statement coverage). One test at each transport boundary proves Retry-After is routed through the bounded policy.

The ceiling itself is asserted against the pure functions rather than end to end, because observing a 30-second wait through an HTTP round trip means a test that sleeps for 30 seconds. That tradeoff is noted in both boundary tests.

Verification

go test ./..., go test -race, go vet, gofmt, and golangci-lint all clean. make coverage passes on all 12 gated packages, including the new one.

Docs

ARCHITECTURE.md gains the package entry and a design rule ("Retry policy lives only in internal/retry"), since this is a cross-cutting invariant new adapters must preserve. README documents the 30-second bound as user-visible behavior.


Follow-up from review

A second commit (27ce6aa) changes what happens when a server asks for a delay beyond the ceiling.

Clamping respects throttling in the wrong direction. A server that asks for a long wait is usually rate-limiting, and a request sent before the requested delay expires can worsen the throttling or extend a ban — so clamping Retry-After: 86400 to 30s means hammering the exact server that asked to be left alone. Neither a 24-hour sleep nor an early retry is acceptable, so the operation now stops with an actionable error:

server asked to retry after 24h0m0s, which exceeds the 30s limit; run the
command again later

A delay the CLI chose for itself is still simply clamped — the ceiling only refuses server-requested delays.

retry.Delay and retry.Wait gained an error return carrying a typed *retry.DelayTooLongError, so the refusal lives in the shared policy and all three call sites (httpapi, webdav, TUS) inherit it with no changes. After no longer saturates at MaxDelay, which would have hidden an excessive hint from the new check; it saturates at math.MaxInt64 and the ceiling decides. Exit-code behaviour is unchanged — both the old 429 error and the new typed error map to general failure.

A test at the webdav boundary asserts the refusal is prompt: one attempt, errors.As matches the typed error, and elapsed time stays under MaxDelay.

@mzner
mzner force-pushed the fix/cap-retry-after branch 2 times, most recently from ec01f61 to d957b5a Compare August 9, 2026 14:32
mzner and others added 3 commits August 9, 2026 16:39
A retried response could dictate how long the CLI paused. Both the WebDAV
and API transports parsed Retry-After and slept for exactly that long with
no ceiling, so `Retry-After: 86400` suspended a single retry for 24 hours.
No request is in flight while waiting, so --timeout could not recover from
it: that value is the per-request http.Client timeout, not a deadline for
the whole operation. A hostile or misconfigured server could therefore hang
any command indefinitely, and the exponential backoff had the same gap for
a large --retries with a long base wait.

Extract the retry policy into internal/retry and apply one ceiling of 30
seconds to every wait, whether it came from Retry-After or from backoff. An
excessive hint is clamped rather than discarded, so throttling is still
respected; the retry just happens no later than the ceiling.

The policy previously existed as three functions duplicated verbatim in
internal/httpapi and internal/webdav, which is why a bound added to one
would not have covered the other. Both packages now call the shared
implementation, and the TUS upload path inherits it.

Retry-After parsing follows RFC 9110's two forms. Previously the header was
parsed by appending "s" and calling time.ParseDuration, which accepted Go
duration syntax the specification does not define ("5m3" became 5m3s) while
rejecting other values outright ("1h"). Delta-seconds now parses as an
integer, and an out-of-range value saturates so that an absurd hint is
capped instead of being read as "no delay requested".

Tests cover the ceiling, both header forms, unusable values, and backoff
growth, plus one test at each transport boundary proving Retry-After is
routed through the bounded policy. The ceiling itself is asserted against
the pure functions, since observing it end to end would mean a test that
sleeps for it.
Clamping a server-requested delay to the ceiling bounded the wait but
sent the follow-up request sooner than the server asked for. For a
legitimate 429 that is the wrong trade: retrying before the requested
delay expires can worsen the throttling or extend a rate-limit ban, and
repeating it every attempt makes that likely.

A server-requested delay is now honored exactly when it fits within the
ceiling and refused with a typed error naming both the requested wait and
the local limit when it does not. Waiting out a 24-hour delay is still
never an option; the command stops promptly instead, and the user can run
it again later. Exponential backoff is the CLI's own choice, so it stays
clamped rather than refused.

After also stops saturating at the ceiling, since the error has to be
able to report what the server actually asked for.
- return an already-canceled context before evaluating retry policy
- cover cancellation combined with an excessive Retry-After delay

Signed-off-by: Matteo <mzner@pm.me>
@mzner
mzner enabled auto-merge (squash) August 9, 2026 14:39
@mzner
mzner force-pushed the fix/cap-retry-after branch from d957b5a to be06f62 Compare August 9, 2026 14:40
@mzner
mzner merged commit c1855b1 into main Aug 9, 2026
14 checks passed
@mzner
mzner deleted the fix/cap-retry-after branch August 9, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant