Skip to content

Latest commit

 

History

History
613 lines (527 loc) · 57 KB

fleet-report.md

File metadata and controls

613 lines (527 loc) · 57 KB

Fleet self-reporting (governor.fleet_report)

A hive can report its own hive-attributable failures upstream to hivecommons/hive, so a defect that shows up across many hives gets noticed without every operator filing the same issue by hand.

Because this files issues on a public repository from your hive, it is off by default and the dry-run preview is designed to be read before you turn it on. This page is the reference for what it sends, when, and what it never sends.

The opt-in

governor:
  fleet_report:
    enabled: true              # tri-state; unset = hosted on, self-hosted off
    repo: hivecommons/hive     # default
    max_per_day: 5             # cap on NEW issues per 24h (joins never count)
    file_upstream: false       # legacy alias for enabled: true

enabled is a tri-state (#11868):

  • unset (the default) resolves to on for hub-managed hives — those booted with HIVE_HEARTBEAT_KEY by the hub, which run under the hivecommons-hive GitHub App and so can write to hivecommons/hive — and to off (dry-run) for self-hosted hives whose PAT is scoped to their own repos;
  • enabled: false forces dry-run everywhere;
  • enabled: true (or the legacy file_upstream: true) forces filing.

Dry-run means the hive evaluates everything, shows you the reports it would file on the dashboard, and writes nothing upstream. The publish path returns immediately when dry-run is set, before any GitHub client is touched. There is one knob, and it is all-or-nothing: there is no per-trigger opt-in. The same settings are exposed in the dashboard under Settings → Governor → "Self-Reported Issues" (owner role) and over GET/PUT /api/config/governor/fleet-report.

The three triggers

A report is only ever built from evidence marked attributable — evidence the hive believes points at hive's own behaviour rather than at your repos, your prompts, or your infrastructure. Non-attributable evidence never produces the report under a trigger.

acmm-shortfall

Fires when an ACMM criterion has been unmet for two or more weekly epochs (PersistentUnmetEpochs = 2) and there is attributable evidence to explain it. The two-epoch requirement means a single bad week never files anything.

The point of this trigger is the pairing: "this hive cannot reach an ACMM bar, and here is the hive-attributable reason why". A shortfall with no attributable evidence is treated as an operator concern instead — it is surfaced to you on the dashboard and is not reported upstream.

Always the self-report tracker, never a standalone issue (#13470). A shortfall is filed in one of two ways, and neither is a Fleet report: issue:

  • When this hive already has an open [selfreport] tracker about the same component (a runtime-error entry, or an earlier shortfall that opened the tracker itself — open, not recovered, not found closed), the shortfall posts one short comment there — "ACMM shortfall rolled up from (): <component> () is blocking criterion <criterion> — " — and its state entry points at that tracker. The roll-up is re-posted only when its text changes.
  • Otherwise the shortfall opens the tracker: a [selfreport] acmm-shortfall: <component> blocks <criterion> issue with the same identity rows, owners, spoke link, Affected hives block and (when the evidence carries any) scrubbed diagnostics a runtime-error report has, plus an ## ACMM shortfall section naming the criterion. A second criterion blocked by the same component then rolls up onto it. When the tracker is closed the next persistent shortfall opens a fresh one; a fingerprint closed upstream inside the 7-day cooldown is not re-filed.

The standalone Fleet report: <component> blocks <criterion> [<fingerprint>] format is retired. Nothing on a current build renders it, the hub relay refuses it from a spoke that still does (below), and the hive that watches the report repo closes any that still land.

Retired: hive-code-defect and every standalone Fleet report: issue

Until #13470 a third trigger synthesised "agent stall", "agent crash loop" and "CI-poll nudge" evidence from bare dashboard counters (StallNudges, Restarts, CIPollNudges) and filed it as Fleet report: … hive-code defect issues with no identity, owners, spoke link or diagnostics — nobody could act on them (#13465 is the canonical example; over 14 days 12 of 18 such reports were closed by the bot itself and about a third reached a fix, against about 82% for self-reports). Those conditions are now runtime-error classes (agent_stall, agent_crash_loop, ci_poll_nudge, below), observed at the moment the agent manager or watchdog sees them, with the kick/pane context attached. A persisted state entry for the retired trigger is pruned on the next evaluation; nothing is filed or recovered under it.

A spoke still running a pre-#13470 build (#13567 came from 5.186.4-22-g20cec0241) can still ask for a standalone issue. Two guards make sure it never gets one:

  • Hub relay. POST /api/selfreport/ingest recognises the retired Fleet report: title. With no open issue for the fingerprint it looks for the relaying hive's open [selfreport] tracker (open, hive/selfreport, body carrying that hive's <!-- hive:selfreport … hive=<id> --> marker) and folds the payload onto it as one comment (the original report inside <details>), answering 200 {"joined": true}. With no tracker it answers 410 Gone {"reason": "legacy_standalone_retired"} and logs the spoke's version; the spoke records an upstream-write failure and retries next cycle, which is a log line, not an issue. Neither path creates an issue.
  • Governor sweep. A spoke whose own credential can write to the report repo files directly and never touches the hub, so the hive that watches hivecommons/hive runs an hourly sweep: one search for open issues titled Fleet report: authored by a Bot account and carrying the <!-- hive-fleet-fingerprint:… --> marker, each closed as not_planned with a comment pointing at #13470 (at most 5 per pass). An issue a person reopened (state_reason: reopened), a human-authored one, or one without the marker is left alone.

runtime-error

Unusual runtime errors the hive notices in itself. "Unusual" is an explicit allow-list of classes defined in pkg/fleetreport/runtime.go; a condition only becomes a report once it crosses that class's threshold, and the report title is [selfreport] <class>: <short message>.

Class Fires when Severity
restart_storm the restart-storm breaker pauses an agent. Message carries the pause reason (restart-storm: <cause>); diagnostics add the restart count inside the breaker window against the pause threshold, the recent restart reasons with their counts (at most five, remainder summarised), and the agent's last error when it differs from the cause. Everything is bounded and scrubbed before storage high
kick_failed_repeatedly ≥3 consecutive undeliverable kicks for one agent after the busy-no-activity stall restart has been attempted (or that restart itself errored) — i.e. the recovery did not bring the session back; kicks that fail before the restart are a dashboard/log warning only medium
panic_recovered a dashboard handler recovered from a panic high
model_discovery_degraded model discovery has been failing for >1h (excluded, audit-log only: copilot serving the HTTP catalog because the SDK helper is absent (#11868), and copilot Not authenticated because the operator has not completed the dashboard Copilot Login (#12947), and copilot not licensed to use Copilot 403s, a seat problem on the reporting hive (#13206)) medium
inference_auth_failed 401/403 from an inference backend (LiteLLM, Copilot) high
inference_backend_auth_failure the inference proxy repeatedly logs inference backend auth failure after an upstream 401, including after a retry with a rotated key when applicable. Check the configured backend credential and the local log's key_source and key_changed_since_start; renew or correct the credential without sharing it. Log-fed: ≥5 warnings in 10 minutes, clears after 20 quiet minutes. This identifies an operator credential failure, not a confirmed Hive code defect high
advisory_post_failed the advisory digest could not be posted (rate-limit skips excluded) medium
agent_stall ≥3 consecutive "continue" nudges sent to one agent after a kick without the pane moving; a credited stall recovery (the pane moves before the next kick) clears the streak. Message carries the nudge count, last kick time and frozen pane hash; diagnostics carry agent, lane, backend, counters and a scrubbed 15-line pane tail (#13470, was the hive-code-defect "agent stall") medium
agent_crash_loop the watchdog latches an agent as crash-looping and pauses it (acting mode only — observe mode would re-count every sweep); the healthy-window reset clears it. Diagnostics carry the classifier detail, failure count/cap, backoff ladder and healthy-reset window (#13470, was "agent crash loop") high
ci_poll_nudge the harness sent a non-monitoring agent the stop-polling-CI nudge (at most one per kick); reviewer, CI-maintainer and quality lanes are exempt because reading CI status across PRs and repositories is their assigned work. Only a command line run more than once since the kick counts as polling (#14016): a sweep that reads each of several PRs' checks once does not cross the count threshold. A self-report fires only after an eligible agent ran ≥2 such repeated polls since the kick — a single nudged poll or a sweep does not trigger a report. One per 24h window; message carries polls since kick / time since kick / nudges since boot, diagnostics the scrubbed pane tail plus the repeated CI-status command lines with their counts (#14289; #13470, was "CI-poll nudge") low
issue_request_stuck an issue/comment/label request file in the relay queue (issue-requests/) has failed ≥3 attempts or has been retrying for ≥30 min. Key is the request file name; message carries kind, attempts and age, diagnostics the last error (scrubbed). Clears when the request completes or is quarantined (.failed/.bad) medium
effect_ledger_op_stuck the mutation boundary refused a retry because the journaled external effect (issue create, PR open, …) has been Unknown/Planned for ≥30 min — the caller keeps replaying the same logical op and it never resolves. Key is the op's logical ID; message carries kind, status and age, diagnostics the attempt count, repo and claim target. Clears when the op is applied, replayed as already-applied, reconciled, or — for issue_create — expired to NotApplied once a refused retry finds it unresolved past 1h (the caller's next attempt re-runs its duplicate search, then creates; #17693) medium
agent_dir_setup_failed the harness failed to set up the per-agent sandbox directory structure — create XDG directories, restore or tighten directory permissions on interactive/inference home directories. Log-fed: keywords include failed to create per-agent xdg dir, failed to restore per-agent dir mode, failed to restore interactive home mode, failed to tighten per-agent dir mode, failed to tighten interactive home mode, failed to restore inference home mode, failed to tighten inference home mode (crux/agent/manager.go), repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
agent_pane_timeout the agent-readiness wait timed out and the harness dumped the tmux pane (prompt timeout — dumping pane / pane content at timeout, pkg/agent/manager.go), and the startup kick dropped because the CLI never became ready or reached its input prompt (startup kick dropped: …, crux/agent/manager_kick.go, the same event as the pane dump). Log-fed: same ≥5-in-10-min repeat threshold and 20-min idle clear as the catch-all below; pane excerpts are scrubbed and bounded before storage low
agent_login_required the login scanner saw the agent CLI asking for /login (login required detected, pkg/loginscan) — the agent's auth lapsed and it is paused until an operator re-authenticates. The record carries backend (which provider to re-authenticate), login_hint (the CLI's login command) and credential_valid; re-login in the agent's tmux pane, then Resume. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
pr_request_head_unpushed the pr-request watcher found the PR head branch missing on the remote (head branch is not on the remote): the agent's push failed, never ran, or went to another ref — the line's checked and push_outcome attributes say which. Logged once per request (retries stay at INFO), so the repeat threshold counts distinct unpushed requests, not one request's retries (#16969). Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
bead_fact_ingest_failed the knowledge system failed to ingest a bead fact from an agent (failed to ingest bead fact, pkg/knowledge), repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
feedback_issue_refresh_failed the dashboard's feedback-issue watcher failed to refresh or update a feedback issue (feedback: issue refresh failed, pkg/dashboard), repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
github_app_repo_not_covered the GitHub App recheck (state=repo-not-covered) found the installation does not include every configured repo — an operator configuration condition (tick the repos in the App's repository access), not a hive defect. Log-fed: same ≥5-in-10-min repeat gate and 20-minute idle clear as agent_pane_timeout low
github_app_alerts_permission_missing the sec-check scoped token mint was refused with Dependabot alerts read requested and retried without it. The App owner must grant Dependabot alerts read-only and re-accept it on the installation; the fallback token cannot read alerts. Log-fed: ≥5 warnings in 10 minutes, clears after 20 quiet minutes. See GitHub App setup medium
github_app_statuses_permission_missing GitHub answered the queue action's prow/lgtm commit-status write with 403 (could not set the prow/lgtm commit status: the GitHub App lacks the statuses: write permission, pkg/github/client.go): the App installation lacks Commit statuses read & write, so Prow may strip the lgtm label and the auto-merge queue may not hold. The App owner must grant Commit statuses read & write and re-accept it on the installation. Only the 403 carries this wording — any other status failure (5xx, 422, transport) logs the neutral could not set the prow/lgtm commit status and is not this class. Log-fed: ≥5 warnings in 10 minutes, clears after 20 quiet minutes. See GitHub App setup medium
hub_heartbeat_rejected_unprovisioned the hub answered the heartbeat 403 hosted/saas hive IDs can only be created via provisioning — it does not know this hive ID (deprovisioned, renamed or registry drift), so retrying cannot succeed. The hive backs off 10 minutes between probes; fix by provisioning or re-IDing the hive. Log-fed, fires on first sighting, 3-hour idle clear medium
github_app_not_installed a per-agent scoped-token mint was attempted while the GitHub App has no installation ID (installation_id 0): the App is not installed on the hive's org, or discovery never found it. An operator configuration condition — install the App or set github.installation_id — not a hive defect. Key is the agent name; reported on the first observation and cleared by the next successful mint. The runtime logs the matching WARN once per agent instead of every supervisor tick (#14186) medium
forge_unreachable a per-agent scoped-token mint got no answer from the GitHub host (dial timeout, connection refused, context deadline — isForgeTransportErr in crux/agent/manager_tokens.go): egress from the pod to the forge is broken. Nothing about the App key or installation follows from it, so it is reported apart from github_app_token_mint_failed and clears that class if it was open. Key is the agent name; cleared by the next successful mint. Fix cluster egress, DNS or the proxy/hostAliases in front of the forge, not the App medium
ioscan_classifier_unavailable the ioscan model classifier failed closed (ioscan classifier failed closed; text withheld, pkg/scheduler/ioscan_enforce.go), withholding untrusted text, or a genuine backend error under ioscan.classifier.fail_mode: open passed text on unclassified (ioscan classifier failed open; text passes on unclassified). The HTTP model endpoint retries one transient failure (429, 500/502/503/504, or a connection dropped before a reply; not a timeout) before either fail mode applies, so a single gateway blip is neither withheld nor passed on unclassified. Under fail-open, a missing model endpoint instead logs ioscan classifier not configured; semantic scan skipped (fail-open); semantic scanning is skipped and this operator configuration condition is not fleet-reported. A missing endpoint under fail-closed still withholds text and is reportable. The scheduler logs ioscan fail-closed blocked kick (pkg/scheduler/templates.go) or ioscan fail-closed blocked held-PR coordination kick (pkg/scheduler/policy_overlays.go), with the classifier_error, when it withholds a kick for the same reason, which maps to this class too. A kick withheld while the classifier is answering — untrusted text tripped the injection policy (a Critical finding or a classifier block verdict) — is the defense working as designed: it logs ioscan fail-closed withheld kick on an injection finding (or ... held-PR coordination kick on an injection finding), stays in local logs and the audit trail, and is not fleet-reported (#18459). The WARN is already deduped per failure by warnOnce (once per process), so it is reported on the first observation (MinCount 1) with the normal idle clear; this is a real degradation, not noise. Log-fed medium
github_fetch_transient a non-critical GitHub fetch failed and the feature degrades gracefully (channel commit date: fetch failed — the row will render without a timestamp, pkg/hub/channel_commit_date.go) — typically a rate limit or transient API error, not a hive defect. Reported only when persistent: log-fed, same ≥5-in-10-min repeat threshold and idle clear as agent_pane_timeout. The sibling SHA poll: commit message fetch non-200 has its own class, sha_poll_commit_message_unavailable low
label_provisioning_failed provisioning the issue stage labels on a repo failed (issue stage labels: provisioning failed, pkg/github/issue_stage_labels.go) — typically the App lacking label permission on that repo or a transient API error; diagnostics carry the repo and scrubbed error. Log-fed: same repeat threshold and idle clear as agent_pane_timeout low
agent_cli_restart_loop the agent CLI crashed (bare shell) or sat on a consent screen and the harness restarted it before the kick (agent CLI crashed or stuck on consent screen, restarting before kick, crux/agent/manager_kick.go, kick_async.go), repeated ≥5 times in 10 min — a restart loop, not transient noise. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
spek_stream_unavailable a Spektacular live meeting child (spektacular meeting, pkg/spektacular/stream.go) keeps exiting and is restarted with exponential backoff ([spektacular] stream child exited; restarting), or gave up after its restart cap ([spektacular] stream child keeps exiting; giving up). The restart line is a WARN at most once per backoff window and INFO otherwise. A spektacular release without the meeting command (before hivecommons/spektacular#126) is not this class: the stream probes spektacular meeting --help once per installed binary, logs one INFO and never starts or restarts the child, and the Meetings panel says notes and clarifying questions are off (#18630). Log-fed: same repeat threshold and idle clear as agent_pane_timeout low
copilot_cli_hung a Copilot agent showed none of the CLI's startup chrome (input prompt, help hint, login request) 180 s after launch, outside kick delivery, so the hang detector killed its session, ran bare copilot to capture the error and relaunched the agent (copilot hung with no CLI prompt, running diagnostic, crux/agent/manager_poll.go), or that diagnostic saw no output before its own deadline and relaunched anyway (diagnostic: timed out waiting for copilot error output, crux/agent/manager_credentials.go; the line carries the pane's output_tail). The key is the message, so ≥5 in 10 min counts every Copilot agent on the hive together: its Copilot agents keep failing to reach their prompt. Typical causes are a stale Copilot token the CLI hangs on (the diagnostic restores or clears it when it sees an auth error), a blocked network path to the Copilot API, or a CLI build whose startup screen the detector does not recognise (cliReadyIndicators, crux/agent/pane_classify.go). Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
proxy_connect_dial_failed the hive proxy could not dial the upstream host of a CONNECT request (proxy: CONNECT dial failed, crux/proxy/github_proxy_connect.go) or of a Copilot usage-sniffed connection (copilot sniff: upstream dial failed, crux/proxy/github_proxy_copilot.go) — typically DNS or network unreachability of an agent-requested host (for example a dial timeout to telemetry.enterprise.githubcopilot.com), repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
proxy_plain_forward_failed the proxy could not complete an agent's plain-HTTP upstream request (proxy: plain HTTP forward failed, crux/proxy/github_proxy_connect.go), returning HTTP 502. Check the requested host and logged error, including DNS and outbound connectivity; this differs from a destination refused by the proxy policy. Log-fed: ≥5 occurrences in 10 min, clears after 20 quiet minutes medium
proxy_tls_handshake_failed the hive proxy's TLS handshake with an upstream host failed (proxy client tls handshake failed, transparent proxy tls handshake failed, copilot sniff: client tls handshake failed, crux/proxy/github_proxy_connect.go, crux/proxy/github_proxy_copilot.go) — typically a certificate validation failure or encryption mismatch on an agent-requested host, repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
review_evidence_write_failed the github-client failed to write a review evidence bundle to persistent storage (review evidence: could not write bundle, pkg/github), repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
selfreport_cleared_comment_failed the fleet report publisher failed to post the "condition cleared" comment on a self-report issue (fleet report: cleared comment failed, pkg/github), repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
inference_caller_unidentified the inference translator refused a caller whose identity could not be established (inference request from unidentified caller refused, crux/proxy/github_proxy_inference.go). The request remains rejected with HTTP 401; this is distinct from backend credential failures. Check the per-agent UID map and socket attribution for the calling process; the log carries the remote address, method and path. Log-fed: ≥5 refusals in 10 min, clears after 20 quiet minutes medium
inference_no_route the inference translator identified the calling agent but had no inference route for it (inference no route, crux/proxy/github_proxy_inference.go); the request is answered with HTTP 502. The log's reason attribute separates the two causes: cleared (a late request from a lingering pane process after the route was cleared on agent stop) and never_set (the agent runs on a backend that does not install a route but still has ANTHROPIC_BASE_URL pointed at the translator). Its job is to keep this WARN out of unclassified_error_repeated. Log-fed: ≥5 occurrences in 10 min, clears after 20 quiet minutes medium
inference_upstream_error a failed or non-2xx response from the configured inference backend (LiteLLM, vLLM/llm-d) on the Anthropic-to-OpenAI translator or MITM reroute path (inference upstream error / inference upstream failed), or a request the translator could not translate (inference translate request failed, logged at ERROR by logTranslateFailure), all in crux/proxy/github_proxy_inference.go, repeated ≥5 times in 10 min — covers transport failures and statuses other than the dedicated 401/403 auth signal. A request refused because the agent has neither an agent model nor a gateway default_model logs inference model not configured; request refused at WARN instead and is not counted here (a benign configuration condition; see below). Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
proxy_plain_forward_refused the hive proxy refused a plain-HTTP forward to a non-globally-routable destination (proxy: plain HTTP forward refused, crux/proxy/github_proxy_connect.go) — a client ignored its proxy bypass for a private address (e.g. the Docker bridge gateway). Log-fed: same ≥5-in-10-min repeat threshold and 20-minute idle clear as agent_pane_timeout. Operator diagnostics: fix the client's NO_PROXY or target address; the refusal is counted as a proxy violation low
sha_poll_commit_message_unavailable the SHA poller's commit-message fetch got a non-200 from GitHub (SHA poll: commit message fetch non-200, pkg/hub/saas_sha_poller.go). The poller keeps serving the SHA without a message. A throttled 403/429 is treated as client-wide: every SHA is parked until Retry-After / X-RateLimit-Reset elapses and a single WARN is logged per hour per status (warnGitHubFetchOnce, key sha-msg:throttled:<status>), so a rate limit or revoked token cannot trip this class. Other statuses (e.g. 404 for a force-pushed-away commit) back off per SHA and are deduped to once per hour per (SHA, status), so one stuck commit logs one line; the class is reported only when ≥5 distinct SHA/status pairs fail within 10 min (many branches whose commits GitHub no longer serves). Its main job is keeping the residual WARN out of unclassified_error_repeated. Log-fed: same repeat threshold and idle clear as agent_pane_timeout low
agent_lifecycle_repeated known agent-lifecycle log lines (failed to send kick, agent CLI crashed (bare shell detected), tmux kill-session before restart failed, agent tmux session missing, watchdog would restart dead agent in observe mode) repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as agent_pane_timeout; replaces the catch-all unclassified_error_repeated for these lines low
tmux_session_check_failed the agent's tmux session has-session check failed (tmux has-session failed, crux/agent/manager.go) — typically the session was lost or the tmux server is unhealthy, repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as agent_pane_timeout medium
transcript_scan_slow a Copilot transcript activity scan took longer than its interval (transcript activity scan outlived its interval, crux/agent/busy_visibility.go) — agent liveness reads may lag. The WARN itself is debounced at the source (transcriptScanPace, crux/agent/transcript_scan_pace.go): it fires once per slow episode, only after 3 consecutive passes outlive the interval, and re-arms only after a pass finishes under 80% of the interval, so one sustained overrun logs one line and never trips this class; it is reported only when ≥5 distinct slow→fast→slow episodes occur within 10 min. Log-fed: same ≥5-in-10-min repeat threshold and 20-minute idle clear as agent_pane_timeout. Operator diagnostics: check the agent's transcript/session volume; it is bounded by concurrency, so repeated episodes indicate slow disk or very large session directories low
kick_freshness_withheld a kick was withheld because every issue/PR ref failed the pre-kick freshness check (kick withheld: every ref failed the freshness check, cmd/hive/kick_freshness.go). The kick is retried on the next cycle; reported only when the same withholding repeats ≥5 times in 10 min (refs persistently unreadable). Diagnostics count stale snapshots, closed items, deferred reads and other read errors for the last withheld kick, without including refs or raw errors. See local per-ref logs for details. Deferred API-budget refusals, refs in a stale repo and refs the re-read found closed or merged log at INFO, so a kick withheld only by those never trips it; a stale repo is reported where it is produced (forgestate: repo cache is stale; serving last-good items (marked stale), GitHub refused repo enumeration transiently; serving last-good snapshot for this repo (marked stale), both WARN in pkg/github/client.go). Its main job is keeping the residual WARN out of unclassified_error_repeated. Log-fed: same repeat threshold and idle clear as agent_pane_timeout low
open_pr_guard_label_failed the pre-dispatch open-PR guard refused a claim but could not put has-open-pr on the issue (open-PR guard: could not label issue / could not ensure label, pkg/github/open_pr_links.go). The label is a tracker signal only: the claim is refused either way. Each issue is retried at most once per 15 minutes after a transient failure (timeout, rate limit, 5xx); a refusal (403/404/410/422: no write access, issue gone or locked) is remembered for 6 hours and logs at INFO, so it never feeds this class. Log-fed: ≥5 WARNs in 10 minutes (#18589) low
models_endpoint_empty the dashboard's inference model discovery (/api/inference/{backend}/models) found no models from any configured endpoint (no models found from any endpoint, pkg/dashboard/api_models.go) and fell back to the static aliases, flagged unverified in the UI. The source logs it at WARN once per backend per hour (DEBUG on repeats, re-armed when discovery succeeds), so this fires only when the condition flaps ≥5 times in 10 min (gateway key rejected, endpoint down). Log-fed: same repeat threshold and idle clear as agent_pane_timeout low
unclassified_error_repeated generic catch-all: the same WARN/ERROR log line (numbers, hashes, paths and quoted strings collapsed) repeated ≥5 times in 10 min. Fed by a slog.Handler tee (pkg/fleetreport/slogfeed) on the hive's root logger, so a repeating defect is noticed even with no hook site. Skips benign lines (read/keepalive i/o timeouts, WARNs whose error is a GitHub rate-limit refusal or a 502/503/504 gateway failure — the caller retries next tick, context cancellation, the permissions watcher's fixed … self-healing lines) and any message that already matches another class's keywords. Clears automatically after 20 quiet minutes (2× the window), so a "cleared" comment follows once the symptom stops low

The cancellation exclusion includes structured error attributes, including logger-bound attributes: typed errors wrapping context.Canceled, exact context canceled strings, and strings ending in : context canceled. Warnings remain in local logs; deadlines and other errors remain reportable, including when another error attribute accompanies a cancellation.

When a client hand-over or shutdown cancels an in-flight contributor-lane automerge pass, the pass stops without warning or stage reconciliation. Other contributor sweep failures remain reportable.

The GitHub rate-limit exclusion also covers the local REST reserve guard: when optional issue activity or CI trail fetches are shed below the reserve, their warnings remain in local logs but do not create repeated-error reports. Issue activity keeps its last rollup and is retried by the collector.

Canceled maintainer-edit access reads during an in-place adoption sweep stay in local logs without triggering the catch-all; adoption is preserved and the read is retried on the next sweep. Other access-read errors remain reportable.

Callers that fail closed on a GitHub HTTP 500 from the combined-status endpoint — the merge-eligible lane gate, the trusted-author and contributor-lane automerge sweeps, and the stage-label check digest — also stay in the logs without triggering the catch-all (failClosedStatus500, pkg/fleetreport/slogfeed). The PR is retried next tick; other errors on those log lines and HTTP 500s elsewhere remain reportable.

The queued, self-authored, trusted-author and contributor-lane automerge sweeps also skip reporting when reason=base-refresh:merge-guard accompanies destination merge already in progress (automergeMergeGuardSkip, pkg/fleetreport/slogfeed/slogfeed.go, #17400). This is expected contention with an in-flight destination merge, retried next tick. The warning stays in local logs; other skip reasons and errors remain reportable.

The generic feed also skips the exact review pending expired message: it records the review planner's bounded retry, and a burst across unrelated PRs must not become a repeated-error report. The original expiry log and its diagnostics remain available. review dispatch exhausted; holding PR for a human still qualifies for reporting when repeated.

It also skips the constant policies: seeded policy differs from the embedded default with no seed record… warning (ReconcileSeededDefaults, pkg/policies/seed_reconcile.go, #13825): a hive whose policy directory predates the seed-manifest tracking added in #9428 has every embedded template diverge on its first post-upgrade reconcile — up to one WARN per template in the same boot — and a burst across unrelated templates is not a repeated runtime defect (#14005). The per-template warning stays in the container log, already deduped per path per process.

It also skips copilot CLI substituted the configured model (logCopilotModelFallback, src/crux/agent/manager_poll.go): each copilot agent warns once per (requested, running) model tuple (#13659) and shows the substitution on its dashboard LastError, so a hive whose copilot agents share one model id the account's Copilot CLI rejects emits one WARN per agent in the same boot. That is an operator configuration condition — change the agents' model — not a repeated runtime defect (#14020).

It also skips inference model not configured; request refused (logTranslateFailure, src/crux/proxy/github_proxy_inference.go): the inference translator refuses every request from an agent that has neither an agent model nor a gateway default_model, and names that remedy on the agent's dashboard LastError. That is an operator configuration condition — select a model or set the gateway default in Model Gateways — not an inference_upstream_error (#16170). Other translation failures still log inference translate request failed at ERROR and stay under inference_upstream_error.

It also skips inference agent answered kick with prose only, sending action nudge (src/crux/agent/manager_kick.go): the prose-only watchdog sends at most one action nudge per kick and counts it in the agent's ActionNudges, so the WARN records a recovery action, not its failure. Repeating this notice across kicks is not a repeated runtime defect (#16795). The WARN and its minutes_since_kick / total_action_nudges fields stay in local logs; actual nudge-delivery failures remain eligible for reporting when repeated.

It also skips stage transition suppressed by dampener (stageDampenerSuppression, pkg/fleetreport/slogfeed/slogfeed.go): the stage-label store suppresses flapping transitions during its dampener hold window (#17103). That is the dampener working as designed, not a runtime error. The same function skips stage flap suppressed (suppressStageFlap, pkg/stageengine/writer.go): the stage engine refuses a same-head, same-evidence reversal of its last label write inside its flap window (#17142).

It also skips the exact WARN signed-commit reconciler: skipped; the PR keeps an unsigned commit (signedCommitOwnershipSkip, pkg/fleetreport/slogfeed/slogfeed.go; emitted by pkg/github/pr_signed_reconcile.go): the signed-commit reconciler deliberately refuses to rewrite a merge commit or a commit authored or committed by anyone other than a hive agent, and records the remedy on the PR (noteSignedSkip). Repeated refusals are the ownership guard working as designed, not a repeated runtime defect (#17924). The WARN and its reason stay in local logs; other reconciler failures and the same message at ERROR remain reportable.

The same reconciler logs a failed read (listing open PRs, a PR's commits or its comments) at INFO, not WARN, when the failure is transient: a transport timeout, a deadline, a rate limit, a 5xx or a shutdown cancellation (logSignedReadFailure, pkg/github/pr_signed_reconcile.go). Every such read is retried on the next pass a minute later, so one slow GitHub minute across a few active repos is network weather, not a repeated runtime defect (#18243). A 401, 403, 404 or malformed response stays a WARN.

It also skips the exact WARN proxy request blocked (recordViolation, src/crux/proxy/github_proxy_violations.go): the GitHub proxy refusing a request that the agent's mode, write-surface or issue-shape policy forbids is sandbox enforcement working as designed. The agent receives a denial body, the refusal is counted per agent and audited, and the WARN already fires only once per distinct (agent, method, path), so one agent trying a handful of forbidden endpoints is not a repeated runtime defect (#18230). The WARN and its agent / method / path stay in local logs; proxy dial, certificate and TLS failures remain reportable.

It also skips the exact WARNs provider quota exhausted: withholding agent kicks, provider spending limit: withholding agent kicks and provider spending limit: withholding all but the probe kick (runEvalCycle, src/cmd/hive/main_eval_cycle.go): withholding kicks from an agent whose provider quota is exhausted (#18112) or while the spending limit is clipped (#15809) is backpressure working as designed. The notice repeats every eval cycle until the quota resets and a probe kick clears the hold, so a long hold crossed the repeat threshold on its own (#18239). The WARNs and their withheld / probe fields stay in local logs; failures to gate, probe or deliver kicks remain reportable.

The re-engagement sweep logs re-engagement kick undeliverable; budget not charged at INFO rather than WARN when the cause is shared session pool is busy (deliverReEngagement, src/cmd/hive/main_sweeps.go): a saturated shared CLI session pool refusing another session is backpressure. The kick can be retried on a later sweep and consumes no repair budget, so a burst across PRs is not a repeated runtime defect (#15350). Other undeliverable-kick causes stay at WARN and remain reportable.

Deferred to follow-ups (defined, not yet wired): upgrade_stalled, data_disk_pressure, github_app_token_mint_failed, unexpected_http_5xx_from_hub.

The queue-health classes exist because of a real incident: eleven placeholder request files (repo: "owner/repo", title t, body b) sat in the relay queue and were retried for six hours — 175 identical warnings sharing one effect-ledger op stuck in Unknown — while the allow-list, having no hook for the watcher, reported nothing. The generic feed catches the shape of that failure (one message, many repeats) for any component; the two explicit classes name the queues an operator actually has to drain.

Each report attaches the class, severity, count and window, the normalised message (numbers, hashes, paths and quoted strings replaced by placeholders) and a redacted diagnostics excerpt inside <details>. Secrets are never collected; the diagnostics go through the same scrubber as the other triggers.

observation

Nous detector findings use the observation self-report class and the hive-observation label. The body contains only hive metadata (hive id/slug, project organization, ACMM level, hive version, and cluster_id) plus aggregate evidence and a JSON daily series suitable for a sparkline. Watched repository names, issue numbers, and file paths are not part of this report.

Observation reports never use the spoke's GitHub client. A hub-managed spoke posts them to POST /api/selfreport/ingest; the hub forces the destination to hivecommons/hive, adds the configured needs-decision label, and searches the open issue body for the detector fingerprint before creating or commenting.

Exactly what leaves the hive

Each report — acmm-shortfall and runtime-error alike, since #13470 — carries:

Field Value
hive the hub-assigned hive id on a hub-managed spoke; the anonymous instance digest otherwise
hive name, spoke hub-managed spokes only (#13211): the hive's display name and a markdown link (name → dashboard URL) — see Which hive; — when unknown
anonymous instance first 16 hex characters of the SHA-256 of your hive ID
owner(s) the GitHub login(s) of the hive's owner(s) — dashboard.authorized_users entries with the owner role plus the device-flow operator — as @login; (anonymous) when include_owners: false
hive version, commit build stamp, short commit
mode the hive's mode
ACMM level e.g. L3
unmet criterion name and key (acmm-shortfall only; also repeated in the ## ACMM shortfall section)
component, agent, lane which part of the hive produced the evidence
error class the classified error, not the raw message
count + window how many occurrences, over what window
detected periodicity if the failures look periodic
repos, owners the owner/repo names this hive watches (project.repos) and the distinct owners derived from them, as N: a/b, c/d — see Repositories and repo owners
repo, ref runtime-error only: the item the error was scoped to, when the class carries one (today advisory_post_failed → the digest's repo); — otherwise
self-recovered no on a report, yes on a recovery

Alongside the rendered body, each report — and the hub relay request to POST /api/selfreport/ingest — carries a structured hive_state block: the hive's 24 h / 7 d state trends from pkg/hivestate (median/p90 queue depth for issues, PRs and held items, SLA-violation rate, agent idle ratio, kicks/day, tokens/day; see Hive state time series). It contains only aggregate counts and rates — no agent names, issue numbers, or repository content — and is kept out of the body, so moving trends never make an open report look changed. It is omitted when the hive has no snapshot in the last 7 days.

Every report carries the same label set: fleet-report, hive/selfreport, origin/selfreport, kind/bug, needs-triage, severity:…, trigger:…, plus class:… for runtime errors and criterion:… for shortfalls. The per-instance component:…/version:…/instance:… labels are gone — the report repo rejected them and the retry used to drop every label (#13465 landed with none). If GitHub rejects a label set the hub relay retries with the core five, then with hive/selfreport alone, never with none. Observation issues additionally carry hive-observation and observation:<kind>.

What it does not send

  • A hive that is not hub-managed sends only an anonymous instance digest; the raw hive id, name and spoke URL never leave it.
  • Observation reports use the hub-provided hive id/slug and project organization metadata required for maintainer triage, but never send watched repository names, issue numbers, or file paths.
  • No raw log lines, error messages, or repository content. Evidence is reduced to a classified error class, a component, and counts.
  • Titles and bodies are passed through logscrub before leaving the hive, which redacts credential-shaped substrings (GitHub tokens, AWS access keys, bearer tokens, JWTs, private-key blocks). This is a backstop, not the primary protection — the report is built from structured fields, not from free text.

Deduplication, comments, and recovery

Every report gets a deterministic fingerprint. A runtime-error fingerprint is a SHA-256 over the error class and the normalised message (numbers, hashes, paths and quoted strings collapsed) — it names the defect, not the build, so the same fault on every version and every hive produces the same fingerprint and a close by a maintainer keeps suppressing it across the daily channel rollout (#14256 was the case where a per-version fingerprint produced one new issue per build). An ACMM shortfall fingerprint covers the class, component, version and criterion key.

Before filing, the hive searches upstream for an existing issue carrying that fingerprint:

  • Found → it adds a comment and a 👍 reaction instead of opening a duplicate. The reaction is how "N hives see this" gets counted.

  • Not found → it opens one issue.

  • Found → the hive adds itself the way a self-report does (#13470): the short Also seen on … comment (not a re-post of the whole body), the hive/selfreport label, and its line in the Affected hives block below.

For runtime-error reports the hive goes further and joins first. If no fingerprint match exists it looks for an open issue labelled hive/selfreport for the same class, then for a human-filed issue whose title or body mentions the class keywords (for example "restart storm", "model discovery degraded", "LiteLLM 401"). The catch-all unclassified_error_repeated class says nothing about the defect, so its class search is additionally scoped to the leading words of the message in the title; with no usable message it does not join by class at all. When any of those matches, the hive adds itself to that issue instead of opening a new one:

  • one comment Also seen on **<hive>** (v<version>, <cluster>) at <time> carrying a <!-- hive:selfreport-seen hive=<slug> fp=… --> marker and its diagnostics excerpt — at most once per hive per 24h;
  • the hive/selfreport and origin/selfreport labels (the per-hive hive/<slug> label is retired, #12539; hive/selfreport is deprecated in favour of the origin/selfreport provenance label and is kept only so existing filters and the joiner's label:hive/selfreport search keep working);
  • a bump to the Affected hives (N) block in the issue body, edited idempotently between <!-- hive:selfreport-affected --> markers: the joining hive gets (or updates) its own line naming its owner(s), and its repos are merged into the block's Affected repos (N) / Affected repo owners (N) union (below).

Who is affected

So a maintainer can tell who is affected and ping them — not just which namespace-style hive name — every hive renders on its own line of the affected block (#12529 follow-up):

<!-- hive:selfreport-affected -->
**Affected hives (2):**
- hosted-aslom (instance b04ca59b7312017e) — owner(s): @alice, @bob — seen 3, last 2026-10-17T11:00:00Z
- hosted-oke-11-r05x — owner(s): (anonymous) — seen 1, last 2026-10-16T19:56:47Z
seen_count: 4
<!-- /hive:selfreport-affected -->
  • owner(s) are the GitHub logins the spoke knows for its owner: dashboard.authorized_users entries holding the owner role (the first entry without an explicit role, plus any :owner entry) and the operator who completed the device flow (/data/gh-user-token). They travel in the report as an explicit owners field; non-GitHub identities (ibmid:…) and anything that is not a plausible login are dropped before they leave the hive. unknown means the hive sent none (a pre-owners spoke).
  • seen N, last <time> are per hive; seen_count stays the total.
  • Privacy. governor.fleet_report.include_owners: false (default true) renders (anonymous) in both the evidence table and the hive's line, sends no logins, and a later report never re-attaches them.
  • When a spoke that predates the field relays through the hub, the hub fills the line from its own record of the hive owner (unless the spoke opted out). Existing issues with the older inline **Affected hives (1):** <hive> list are upgraded to the per-hive form on the hive's next join, so an old issue gains the owner on the next sweep.

Which hive

So a maintainer can tell which hive a report came from — and open it — without translating a hub id through the hives listing, every report carries the hive's display name and dashboard URL (#13211), both in the evidence table:

| hive | hosted-available-oke-260812-0nzc-available-ok-ba44 |
| hive name | kellya |
| spoke | [kellya](https://kellya.hive.hivecommons.dev) |

and on the hive's affected-block line, as <name> (<hive id>) plus an open link cell:

- kellya (hosted-available-oke-260812-0nzc-available-ok-ba44) (instance 2e281aad9b0e1e15) — [open](https://kellya.hive.hivecommons.dev) — owner(s): @clubanderson — seen 1, last 2026-10-20T16:53:31Z
  • Where they come from. The spoke sends its own view (project.name, and dashboard.public_url else the hub-delivered hub.dashboard_url) as explicit hive_name / spoke_url fields. When the report goes through the hub relay, the hub overlays what its hives listing shows — the operator-set project_name (renames land on the hub, not the spoke) and the validated vanity URL — and fills both for a spoke that predates the fields, falling back to the org or primary repo for the name and to the heartbeat-reported dashboard URL, then the placeholder host, for the link. A later join refreshes the line, so a renamed hive shows its new name on the next report.
  • Privacy. Both fields leave the hive only when it is hub-managed — the same condition under which the hive id (rather than the anonymous digest) is named today. A hub-less hive stays anonymous: the rows render —, the affected line is unchanged, and nothing beyond the name and URL is added. Names are reduced to a single line without markdown structure before they leave; URLs must be absolute http(s) URLs and lose any credentials, query or fragment.
  • Older issues and spokes. A report without the fields renders exactly as before (| hive name | — |, | spoke | — |, the pre-existing bullet form); the hub completes the hive's line and rows when it files or joins on the spoke's behalf.

Repositories and repo owners

So a maintainer can tell whether a fleet issue is confined to one project or one org, every report names the repositories the reporting hive serves (#12338). The evidence table carries the hive's own view; the affected block carries the union across every hive that has reported, re-derived idempotently on each join:

<!-- hive:selfreport-affected -->
**Affected hives (2):**
- hosted-aslom (instance b04ca59b7312017e) — owner(s): @alice — seen 2, last 2026-10-17T11:00:00Z
- hosted-available-oke-11-placeholder-r05x — owner(s): @carol — seen 1, last 2026-10-16T19:56:47Z
**Affected repos (3):** acme/widgets, hivecommons/hive, hivecommons/pluk
**Affected repo owners (2):** acme, hivecommons
seen_count: 3
<!-- /hive:selfreport-affected -->
  • Affected repos is the sorted union of every reporting hive's watched repos plus any triggering repo; Affected repo owners (the orgs/users owning those repos — distinct from the per-hive owner(s) logins above) is always re-derived from it, never parsed back. A hive that re-reports with nothing new leaves the body untouched; a known hive bringing a new repo edits only the block.
  • Repo names are validated to exactly owner/name (lower-cased, GitHub-safe characters) before they leave the hive; anything else is dropped.
  • None of this enters the fingerprint. The fingerprint stays class + normalised message + version, so two hives watching different repos still dedupe onto one issue.
  • The lines are omitted entirely (and the table cells render —) when no reporting hive contributed repos, so pre-#12338 hives and opted-out hives produce exactly the block they always did.

Privacy. The report repo is public and hive.yaml does not record whether a watched repo is private, so the hive cannot filter private repos for you. governor.fleet_report.include_repos (default true; Settings → Governor → "Self-Reported Issues" → "Name the watched repositories in self-reports") turns the whole feature off for a hive whose project.repos include private repositories: the table shows —, the report carries no repos, and the hive's joins never add to the union. The hub relay forwards the body as built, so the switch governs relayed reports too.

A new issue is opened only when nothing matches. Rate limits apply to new filings only: at most max_per_day per hive per 24h, at least 1h between new filings of different classes, and a fingerprint that was closed upstream is never re-filed by the same hive while the hive keeps observing the condition — a close by anyone (maintainer, scanner, close-on-merge) is an instruction to that reporter to be quiet about that fingerprint. The 7-day cooldown runs from the later of the close and the last tick that still saw the condition, so it only expires once the condition has been absent for a whole week; a recurrence after that is a regression and gets a fresh issue. The hub relay applies the same rule for relayed reports (a recently closed fingerprint answers closed_upstream and writes nothing). For the scanner and maintainers this means "already fixed on v5" is a safe reason to close a self-report even when the reporting hive's build (named in the body) predates the fix: cite the fixing PR once and close; the reporter will not re-file or comment again, and picks the fix up on its next build. Filing is also deferred while the GitHub API budget is not normal. Hub-managed hives whose credential is refused by the report repo (403/404) relay the report to the hub's POST /api/selfreport/ingest with their heartbeat bearer, and the hub files or joins under the App identity.

The hive also remembers the body hash of what it last posted, so a condition that persists unchanged does not re-comment every cycle. It re-reports only when the body actually changes, when the issue is gone, or when a previously recovered condition comes back.

When the condition clears, the hive posts a recovery comment (self-recovered: yes). If the hive opened that issue itself, it also closes it; if a human opened the issue, the hive comments but leaves it open — it will not close someone else's issue.

runtime-error issues are never auto-closed, even by the hive that opened them: when the condition clears for that hive it posts a cleared on <hive> at <time> comment — at most one per 24h when the condition flaps, and none once the issue is closed — and leaves the issue for a human to confirm the fix.

State

Report state lives at /data/fleet-report-state.json: the per-criterion epoch history that drives the two-epoch threshold, and the open-issue records (number, URL, whether the hive opened it, component, body hash, recovery flag, and for a rolled-up shortfall the fingerprint of the self-report tracker it commented on). Deleting it resets the epoch counters and makes the hive forget which upstream issues are already open, so a persisting condition can file again.

Previewing before you opt in

The evaluation runs on every dashboard status refresh regardless of the knob — dry-run changes only whether anything is written upstream. So you can leave enabled unset (self-hosted) or enabled: false, run normally, and read the reports the hive would have filed on the dashboard's fleet-report panel.

That is the intended way to adopt this: watch the dry-run output until you are satisfied that what it would send is both accurate and something you are willing to publish, then opt in.

See also