device/telemetry: keep probing when the epoch fetch fails

[elitegreg]

Summary of Changes

• Pinger.Tick fetched the current epoch first and returned early on failure, so one unreachable ledger RPC endpoint stopped all TWAMP probing. Probing is pure UDP and needs no ledger access — the epoch only builds the sample buffer's PartitionKey. During the 2026-07-29 outage (#4130) this cost up to 19 hours of latency samples per device across 23 mainnet-beta devices, for probes that would have succeeded the whole time.
• The pinger now caches the last known epoch and falls back to it, so an RPC outage costs epoch precision rather than measurements. Samples buffer through the outage and flush once submission recovers.
• The epoch is refreshed on its own loop instead of inline per tick. This matters as much as the fallback: a failing fetch burns ~130s across its retries and time.Ticker buffers only one tick, so an inline fetch swallowed roughly a dozen probe opportunities per failure. A cached fallback alone would have resumed probing at ~1/13th the sample rate. The refresh cadence defaults to -probe-interval, so the epoch RPC rate is unchanged.
• Probing is refused only when no epoch has ever been fetched, or when the cached one is older than -max-epoch-staleness (default 12h). The bound is load-bearing: contrary to the issue text, stale-epoch samples across a rollover are not discarded — Submitter.Tick removes a past-epoch partition only when it is empty, and the program never validates the epoch against the clock (it is PDA seed material only). They land in the previous epoch's account, where the lake's derived event_ts places them past the real epoch boundary.
• Both refusal cases log once rather than per tick, and repeated fetch failures collapse into fresh→stale and stale→fresh transitions. The per-attempt retry warning in getCurrentEpoch drops to Debug — at a 10s cadence it emitted thousands of lines across the outage, which is what [TRACKER] Telemetry outage 2026-07-29 — ledger RPC wedge and agent resilience #4130's "single-digit log lines per component" exit criterion is about.
• New pinger_epoch_unavailable error type and a doublezero_device_telemetry_agent_epoch_cache_stale_age_seconds gauge, matching the shape of the existing CachingFetcher staleness metric.
• Fixes device/telemetry: pinger aborts all probing when the epoch fetch fails #4125

Testing Verification

• Five new subtests, all under -race: probing continues under the cached epoch while the fetch fails; the fallback and recovery each log exactly once; probing stops past MaxEpochStaleness with an injected clock; nothing is recorded and one error is logged when no epoch was ever fetched; and the probe loop holds its cadence while the epoch fetch hangs forever (the coalescing regression).
• Verified the tests discriminate by mutating epochForTick back to the pre-fix inline-fetch-and-give-up behavior: four of the five failed. The recovery test initially passed under the mutation because pre-outage samples satisfied it, so it now asserts the stale partition keeps growing while the fetch is failing.
• The two pre-existing epoch subtests pass unchanged — the retry-count and give-up-with-no-epoch behaviors are preserved by the inline first fetch.
• Ran the pinger suite three times to check for timing flakiness in the Run-driven tests.
• internal/netns/TestRunInNamespace_EmptyNameErrors fails identically on a clean main (needs privileges for setns); unrelated to this change.

[Copilot]

Pull request overview

Improves resilience of the device telemetry agent’s TWAMP probe loop by decoupling epoch fetching from the probe tick path and falling back to a cached epoch during ledger RPC outages, preventing probe gaps like the 2026-07-29 incident.

Changes:

• Cache the last known epoch in Pinger and keep probing during epoch fetch failures, with a configurable staleness bound and log-once transitions.
• Add a background epoch refresh loop plus new metrics/error typing to make epoch-cache health observable.
• Add targeted subtests covering fallback, recovery, refusal cases, staleness cutoff, and non-blocking behavior under a hung epoch fetch.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
controlplane/telemetry/internal/telemetry/pinger.go	Adds epoch cache + refresh loop; probes use cached epoch and refuse only when unavailable/too stale.
controlplane/telemetry/internal/telemetry/pinger_test.go	Adds subtests validating fallback, recovery, staleness refusal, log coalescing, and non-blocking probe cadence.
controlplane/telemetry/internal/telemetry/config.go	Introduces MaxEpochStaleness config with defaulting in validation.
controlplane/telemetry/internal/telemetry/collector.go	Plumbs MaxEpochStaleness and NowFunc into PingerConfig.
controlplane/telemetry/internal/metrics/metrics.go	Adds epoch-cache staleness gauge and pinger_epoch_unavailable error type.
controlplane/telemetry/cmd/telemetry/main.go	Adds -max-epoch-staleness flag wired into telemetry config.
CHANGELOG.md	Documents the behavioral change and new flag in Unreleased notes.