Repository navigation
Track managed RBE kernel fault causing VM-suite re-execution #16222
Description
The required CI for #16220 passed, but its VM suite had a physical infrastructure failure and automatic re-execution. Preserve this as a rerun-inclusive reliability/latency cost, rather than counting only the final green target summary.
Observed on test invocation 803186fd-16fa-57b5-b4b2-cbc6ae2cd6ff, tested synthetic merge 6f1ead144a1b673179dca7618816d66440f76ab1:
//agentplane/system_tests/suites:test_vm: first physical action ran 2026-11-13 12:09:12.791255–12:11:47.145982 UTC (154.355 seconds), then REAPI code 14, exit -2: VM health check failed withDeadlineExceeded.- Its server log
vm_log_tail.txtshows managed VM kernel5.15.0 #1, general protection fault with a non-canonical address,update_blocked_averages+0x250/0x740, taskkworker/6:1, workqueuevirtio_vsock. This is the action VM's kernel, not the nested Agentplane test guest kernel. - BuildBuddy automatically re-executed the action at 12:11:49.820769–12:17:48.654080 UTC (358.833 seconds), and the target passed. All three required GitHub checks succeeded and the PR merged.
- The invocation has six NEW physical TestRunner records for five distinct targets; one has the infrastructure error. This single run does not estimate the population failure rate.
Raw server-log artifact: CAS SHA256 0734dbf7c87d66c08cd445c450bb72bdda59635d9555bbe104b72b4fa3fbb238, 19,592 bytes. Retrieve via bb download <digest>/19592 after fetching the failed execution response using bb execution get. The response's serverLogs.vm_log_tail.txt supplies the digest. No application source or CI resource policy was changed to mask the failure.
The observed kernel fault narrows the failure below the health-check symptom; its cause, and any relation to snapshot restoration, remain unknown. The failed action reports 666.207 CPU seconds and ~3.20GB peak action-cgroup memory, neither a whole-VM measurement nor evidence of an OOM. Do not raise test timeout/size or label it an application flake from this evidence.
Follow-up / closure evidence:
- Collect this same server-log signature from subsequent physical failures, with failed and recovered attempt counts separate from final target outcomes; connect it to Agentplane CI #10: Refresh backend CI cohort and publish ten-lane follow-up measurements #16034's recurrence denominators.
- Identify the managed executor kernel/snapshot failure owner and available correction. An upstream issue/contact requires a separate authorized communication; none is claimed here.
- Compare a bounded routing or fresh-VM experiment only after measuring retained-baseline setup cost and proving capacity. Do not route the existing managed VM workload into the saturated self-hosted KVM pool on this evidence.
- Verify the actual correction with exact source/executor identity and several busy windows, preserving failures and retries. A green re-execution alone does not close this issue.
Related: #16099 fast-CI architecture, #16034 reliability follow-ups.
Activity
Record native auth contracts and physical CI failure evidence (#16224)
agentydragon commentedon Nov 13, 2026
A fresh recurrence during #16102 architecture validation: native a90059e6-cddf-459f-9928-fbd2d3c65ed1, source 8a555b2 plus the readonly-mount/shared-supervision candidate.
The first physical
test_vmprocess runs 18:52:45.089992–18:55:20.433288 UTC on 2026-11-13 (155.343s), consumes 530.514 CPU seconds, then ends REAPI14/exit-2: Firecracker socket connection refused while pausing. Its retained server log proves the lower-level failure: Firecracker1.16.2 loads/resumes a snapshot, Linux5.15.0 faults atupdate_blocked_averages+0x273/0x740, then panics in interrupt context and reboots. This is the managed action VM, not the nested Agentplane guest. Peak action-cgroup memory 3,033,096,192 bytes does not establish host OOM.BuildBuddy automatically retried the action. The same accepted invocation finished green: the replacement VM process took 525.032s, its full scenario passed, and all three selected targets passed in 722.499s Bazel elapsed. Preserve the failed attempt alongside the green final result. Case exposure before the panic is unknown. Both separate direct guest actions pass the new shared supervision checks; their results do not erase this deployed-action failure.
Server-log digest
a4c52f83ee7ae97157b19ba1bec2e18ff77f75e6eb7c8ad3721d0743a45fac50/24191, obtained from the failed execution's cached response (/uploads/51d06d77-90de-4676-b257-ea68ebb1656e/...). No raw auth/event records are attached.Existing scheduler investigation already records four November2 related failures, including one fresh boot. Restore is therefore not a necessary trigger. Pinned provider source f3508d33 exposes its newer kernel via an internal/server-side experiment, with no client action property selecting that kernel.
allow-remote-snapshots=falsecan still restore local state;recycle-runner=falsecan force cold boot but retains the vulnerable kernel and discards useful setup reuse. No such policy change is justified by a few passing runs.Next experiment requires provider-controlled current/maintained-kernel cohorts with verified per-attempt kernel identity, cold/restored paths, unchanged full VM assertions/resources and snapshot policy. Retain panic counts and process-time distributions; stop on unverifiable cohort identity or the same panic in the repaired cohort. This is source capability and a concrete follow-up, not a deployed fix or rate measurement. No upstream communication has been made.