Skip to content

device/telemetry: over-capacity submitter path discards the whole backlog, not the oldest slice #4149

Description

@elitegreg

When submission fails all attempts, Tick prepends the batch back only while there is room:

overCapacity := s.cfg.Buffer.Len(partitionKey)+len(tmp) >= s.cfg.Buffer.Capacity(partitionKey)
if !success && !overCapacity {
    s.cfg.Buffer.PriorityPrepend(partitionKey, tmp)
}

(controlplane/telemetry/internal/telemetry/submitter.go:246-252)

The first time that check trips, the entire accumulated backlog goes rather than the oldest slice, and accumulation restarts from zero. Two problems follow:

  • The samples that survive are worse than the ones that were lost. The account stores one start_timestamp_microseconds and one sampling_interval_microseconds with a flat samples array (smartcontract/programs/doublezero-telemetry/src/state/device_latency_samples.rs:117-123) and the reader synthesizes each timestamp as start + i*interval (controlplane/telemetry/internal/data/device/latencies.go:205). A dropped run leaves no marker, so every sample appended afterwards is backdated by the full length of the gap.
  • Dropping everything loses far more than it has to. Keeping the newest Capacity - Len samples would bound the loss to what genuinely does not fit.

#4143 bounds -max-epoch-staleness to partitionBufferCapacity × -probe-interval with headroom, so the probe loop stops adding before this path is reached. That closes the route in but leaves the path itself sharp for any other caller or any future capacity change.

Fix: keep the newest Capacity - Len samples instead of discarding all of tmp, and log what was dropped. Note the slice handed to Recycle must stay the original one — reslicing from the middle would return a shifted array to the pool.

Reported by @ben-dz in review of #4143.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions