Harden redis metrics collection against transient errors - #5131

⚠️ Warnings

⚠️ Your PR has warnings. Please review the issues below.

🚦 2 Pipeline jobs failed

CI | All jobs succeeded

View in Datadog · View in GitHub Actions

CI | test / Golang Unit Tests

View in Datadog · View in GitHub Actions

1 failed test. Error: Should be empty, but was [0xa4f02a64200 0xa4f02a64300 0xa4f02a64400 0xa4f02a64500 0xa4f02a948a0 0xa4f02a46080 0xa4f02a460c0] in collector_test.go:337.

Useful? React with 👍 / 👎

Greptile Summary

The PR hardens Redis metrics collection against transient failures while retaining the last successful business-metrics snapshot.

• Retries transient scan failures with bounded exponential backoff and per-attempt timeouts.
• Publishes failure telemetry without clearing the last successful Redis metrics.
• Skips streams that disappear between discovery and inspection.
• Validates retry backoff settings and safely formats wrapped or ordinary configuration errors.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

sequenceDiagram
    participant C as Metrics collector
    participant R as Redis scanner
    participant S as Atomic snapshot
    C->>R: ScanAll with per-attempt timeout
    alt Scan succeeds
        R-->>C: Stream information
        C->>C: Reset and rebuild business metrics
        C->>S: Publish successful snapshot
    else Retryable failure
        R-->>C: Timeout or connection error
        C->>C: Wait with bounded exponential backoff
        C->>R: Retry ScanAll
    else Final or non-retryable failure
        R-->>C: Error
        C->>C: Increment failure telemetry
        C->>S: Republish retained business metrics plus current error telemetry
    end

_{Reviews (11): Last reviewed commit: "Fix redis collection" | Re-trigger Greptile}