Skip to content

Workflow-run GaugeVec labels accumulate beyond the 12-hour window #91

Description

@jmcerda

Summary

v1.9.0 fetches only workflow runs created in the last 12 hours, but the two per-run GaugeVec collectors retain label sets from every prior refresh. In an active installation, expired run IDs disappear from the API response but remain registered in the exporter, so metric cardinality and process memory grow until the process is restarted or reaches its memory limit.

The current master loop sets workflowRunStatusGauge and workflowRunDurationGauge for each returned run, then sleeps and repeats without resetting either vector or deleting expired label values:

https://github.com/Labbs/github-actions-exporter/blob/master/pkg/metrics/get_workflow_runs_from_github.go#L108-L144

Minimal reproduction

  1. Start v1.9.0 with one or more repositories that create workflow runs continuously. Keep the default per-run fields, especially id.
  2. Record these values after a fresh start:
    • count(github_workflow_run_status)
    • count(github_workflow_run_duration_ms)
    • go_memstats_heap_alloc_bytes
    • process_resident_memory_bytes
  3. Let the exporter refresh across several 12-hour windows while new workflow-run IDs are created.
  4. Repeat the measurements after older runs are outside the query window.

Expected: expired run label sets are removed, and series count / heap / RSS remain bounded by the configured workflow window.

Actual: expired label sets remain in the GaugeVec maps. In one three-repository deployment, RSS grew from roughly 45 MiB after restart to about 253 MiB at a 256 MiB container limit over several weeks, without a corresponding increase in the intended 12-hour window.

Suggested direction

Reset the per-run gauges at the start of each collection cycle, or explicitly delete label values for run IDs no longer present in the current window. A maintained implementation of the reset-and-prune approach is visible here:

https://github.com/amirwollman/github-actions-exporter/blob/v1.0.6/pkg/metrics/get_workflow_runs_from_github.go

This report contains no credentials or private repository data.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions