test: remove watcher lock timing flake - #2876
Merged
kunchenguid merged 1 commit intoAug 23, 2026
Merged
Conversation
…artup tests/fm-watcher-lock.test.sh passed in isolation but failed intermittently under full-suite and ambient concurrent load. bin/fm-watch-arm.sh computes its confirmation deadline immediately after forking the real child watcher, so the child's entire fork, exec, lock acquisition and beacon publication has to land inside that wall clock. Two cases shrank that budget to one second, leaving a two-second window for work measured at 3.1-4.9s under CPU oversubscription, so the arm honestly reported "FAILED - no live watcher with a fresh beacon" and their premises collapsed. A third case ran on the production budget, but its child must also execute a registered check before exiting: measured at 1.9-2.3s idle and 9.1-13.1s under load, against an 11s budget. The two cases that must confirm a real child now hold the arm to production's own budget instead of a shrunken fixture one, the immediate-wake case gets an explicit budget with headroom over its measured loaded cost, and the two waits for the arm's typed failure are sized off the largest production default rather than a fixed eight seconds. No bin/ change and no default behavior change: the lock's fail-closed semantics, SIGSTOP handling, stale-heartbeat detection and the arm's typed failures are untouched. Verified 4/4 green at 3x CPU oversubscription (loadavg 75-80) after 3/3 red before the change, and CONTRIBUTING.md records the convention.
Confidence Score: 5/5The PR appears safe to merge; no concrete changed-code defect or security issue was identified. The test-only timing changes preserve watcher fail-closed behavior, return early on successful confirmation, and provide bounded waits that cover the platform-specific production deadlines. Reviews (1): Last reviewed commit: "test(watcher): stop fixture confirmation..." | Re-trigger Greptile |
Owner
|
Speaking as Kun's firstmate: Scheduled 11:10am PT 8/23 pass. VISION.md read in full from current main VISION (inspected Class: corrective (test-only flake removal). Auto-eligible class. Security: none. Test and contributor-docs only. No workflow-file / secret / injection risk. Overlap / HOLD: none of the standing spawn/teardown/herdr/pool holds. Same-file overlap with older open #904 (broader watcher-lock de-flake, MERGEABLE UNKNOWN, last update 2026-08-17, not covering #2844), #2809 / #2796 on CI / NM: HEAD Workflows: already approved (CI completed SUCCESS on this HEAD). Run IDs: 32649070213 (CI), 32649070247 / 32649927904 (Require no-mistakes). No pending first-time-fork approval. Land-eligible rec: YES (corrective test-only; matching NM attestation; green CI; no standing-hold mechanism overlap). Captain-flag NOW: no. |
|
The changes are sound. A few observations on the specifics:
Removing CONTRIBUTING.md — the two added lines are precise and actionable. They capture the actual invariant ("never below what the real work inside that window costs on a loaded machine") without over-prescribing. No issues from this review. The fix is narrowly scoped, causally motivated by the reproduction data, and leaves production behavior unchanged. |
Owner
|
Speaking as Kun's firstmate: this is merged. Thank you @karotkriss — really appreciate you taking the time on this. |
doitdigital0495
added a commit
to doitdigital0495/firstmate
that referenced
this pull request
Aug 23, 2026
* fix(bearings): restore decision options and add close controls (kunchenguid#2707) * fix(bearings): always show decision options and a close/drop control Freeform-only Captain's Call cards hid the option buttons the board was designed around, and there was no way to drop a stale hold without inventing an answer. Require selectable options, keep freeform as a supplement, and route the reserved __drop__ answer through decline so the hold leaves Captain's Call. * no-mistakes(review): Fix drop closure and decision-only option validation * no-mistakes(review): Preserve answerability for non-decision cards * no-mistakes(document): Clarify decision drop documentation * ci: require no-mistakes pipeline step attestation (kunchenguid#2710) Signature-only PRs can hide skipped review, test, or document steps. Fail unless no-mistakes >= 1.46.0 attests those three steps completed. * feat: collapse decisions into tasks held for the captain (kunchenguid#2728) * feat(captain-hold): collapse the decisions concept into tasks held for the captain A decision is no longer a separate type: it is an ordinary backlog task held for the captain, identified by its task id. bin/fm-captain-hold.sh owns the surviving behaviors - guarded hold creation, the recorded-answer close (answer/answers with a release mode for captain-gated work), the source bindings, and the investigation completion gate - and bin/fm-decision-hold.sh becomes a one-release compatibility shim over it. The fleet snapshot now parses hold-until and computes captain_actionable as queued + captain-held + unblocked + due, independent of row kind, plus a presentation-only deferred_marker for prose-deferred rows. Bearings renders every due captain-held task in Captain's Call, date-deferred holds as dated Charted Next gates, suppresses prose-deferred rows from default views with an omitted disclosure, and excludes from Recently Landed anything that closed while still held for the captain. Legacy compatibility: pre-collapse <origin>-decision-<key> rows are already plain task ids and keep working; short keys in recorded metadata, concrete origin bindings, chat --resolve-key fallbacks, and old resolution records all resolve in place. * no-mistakes(review): Fix captain answer replay and body preservation * no-mistakes(review): Fix captain hold idempotency and legacy replay * no-mistakes(review): Validate card close modes and compatibility routing * no-mistakes(review): Enforce release replay mode matching * no-mistakes(review): Prevent duplicate decision cards and released replay mismatches * no-mistakes(review): Preserve answer columns and legacy resolve replays * no-mistakes(document): Document strict replay and legacy compatibility * no-mistakes(lint): Quote done literals to satisfy ShellCheck * no-mistakes: apply CI fixes * fix(rebase): keep collapsed captain hold board semantics * fix: bound recovery announcements and preserve supervision (kunchenguid#2733) * fix(watch): announce recovery once per generation and keep successors supervising A lost Pi/OpenCode handling handshake re-announced the same recovery generation on every cycle and spent the successor's first ~55s blind, so a real crew event could be ignored and then dropped. Record the announcement in the durable marker, confirm the handshake before the follow-up without swallowing failure, and enter the poll loop immediately. * no-mistakes(review): Tighten recovery event timing regression * no-mistakes(document): Document recovery-loop supervision guarantees * fix(bin): surface captain-call record divergence (kunchenguid#2744) * fix(bin): signal a captain call resolved in the log but still held A captain call has two records and closing one has never closed the other: a `resolved [key=...]` line closes the status-log fold, while the backlog task held for the captain closes only through `fm-captain-hold.sh answer`. Answering on the status side alone left no trace of the disagreement - the fold went quiet, the durable record kept saying the captain owed an answer, and nothing warned. The defect was never the separation; it was the silence. Add `fm-captain-hold.sh diverged`, a read-only report of that contradiction, and print it from `fm-wake-drain.sh` as a bounded RECORD DIVERGENCE section beside OPEN DECISIONS on every drain. It flags one condition: a task still open and still carrying the captain-hold annotations whose key was closed on the status side by the resolve verb, under the collapsed identity or the legacy derived one. It closes nothing, ever. A captain call closed wrongly leaves review entirely, which is worse than the noise, so both reconciliation directions stay human-owned and the printed hint names both - a resolution is not proof the captain ruled, since a call can dissolve on a false premise or turn out to have been a question of fact. Three states are deliberately not divergence: a `captain-held` close is the verified transfer `complete` writes, a still-open keyed decision belongs to the OPEN DECISIONS fold, and a captain call with no routed work item is legitimate rather than incomplete, so routed work is no part of the test. `fm-classify-lib.sh` gains `status_key_closing_verb`, which reports how the status side currently reads one key by replaying the existing `_fm_decision_fold_line` rule rather than re-deriving it, so the two closing verbs stay distinguishable in one place. The per-wake cost is one `tasks-axi list`, one key scan per status log, and the precise per-key fold only for a key that already names a still-open task; the call is hard-bounded so a slow backlog tool can never delay wake presentation. * fix(document): Correct divergence lifecycle documentation * fix(document): Neutralize divergence lifecycle prose * fix(bin): re-arm after an abandoned auto-arm claim and defer a wedge escalation while a worktree is written (kunchenguid#2524) * fix(watch): re-arm supervision after an abandoned auto-arm claim A Claude auto-arm cycle that armed, delivered one rewake, and exited left its single-flight lock behind. Both Stop-event participants then deferred to that lock forever, because its recorded pid was still live: the turn-end guard read it as recovery under way and allowed the stop, and the next Stop firing treated it as another owner and declined to arm. On 2026-08-14 a home with two tasks in flight lost supervision for about 40 minutes with no watcher process and no watcher lock, its beacon frozen at the one delivery, and both crewmates' finished reports sat in the durable queue until an operator drained it by hand. Abandonment is now proven from the epoch ledger instead of inferred from pid liveness. A lock whose holder pid matches the ledger's own owner_pid while the recorded outcome is anything other than arming has already finished its decision, so that claim is reclaimed under the lock's steal mutex, stops counting as recovery ownership in the guard, and is cleared by the guard's terminal check rather than deferred to. A failed clear re-blocks instead of allowing a blind stop, and an arming entry stays in flight however old it is, because its owner foregrounds the arm for the whole watcher cycle. Issue kunchenguid#2251's PR kunchenguid#2263 does not cover this failure. It is closed and unmerged, lives entirely in bin/fm-watch-arm.sh, and retires the stalled watcher and matching stale watcher lock of an arm that is currently running. Here no arm and no watcher were running and no watcher lock existed, so it has nothing to retire and the home stays blind. tests/fm-claude-stop-autoarm.test.sh covers the reclaim, the still-arming and unnamed-owner cases that must keep the gate closed, and the failed clear. tests/fm-turnend-guard.test.sh covers the guard side of the same boundary. Both fail without this change. * fix(watch): defer a wedge escalation while the task worktree is written The wedge detector had two inputs, rendered pane quietness and the run step, and neither can see a crew that is writing source, then tests, then documentation behind a static pane. On 2026-08-14 one crewmate produced eight consecutive possible-wedge escalations in a single afternoon, three of them demanding deep inspection, while it was demonstrably working and then committed. Every one of them cost a supervision turn to disprove by hand. Add write activity inside the crew's own recorded worktree as a third liveness input. crew_worktree_written_since compares the worktree against the caller's existing idle-window timer file, so -newer needs no clock arithmetic, no temp file, and no portable mtime write. The probe runs only inside the branch that was about to escalate, which bounds it to one pruned, depth-bounded walk per window per FM_STALE_ESCALATE_SECS and leaves the per-poll stale sweep exactly as cheap as before. Positive evidence defers rather than cancels. The idle timer restarts so the next window probes again, the escalation counter is neither advanced nor reset so a later genuine wedge keeps the demand-deep-inspection history it earned, and a .writing-since marker ages the whole deferral chain so the pane still re-surfaces once per FM_PAUSE_RESURFACE_SECS, through the same throttle shape a declared pause already uses, labeled as a recheck rather than a wedge. This can only reduce false positives: every absence of evidence, including no recorded worktree, a torn-down worktree, a missing anchor, and a failed walk, falls through to the unchanged escalation schedule, so a crew that writes nothing still escalates on the existing timetable. What the signal cannot see, by design or by construction: - CPU burn with no writes, such as a long compaction, is invisible. That case keeps the old behavior exactly. - A commit-only phase writes only .git, which is pruned first so that firstmate's own read-only git commands against the worktree can never make the probe self-fulfilling. - Writes under the pruned generated trees, or deeper than FM_WORKTREE_WRITE_MAXDEPTH, do not count. - The probe cannot attribute a write to the crew, so a background build or another process touching the tree looks the same. The hourly re-surface is what bounds that, and a churny file cannot buy silence. - The away-mode daemon's own escalation path is deliberately untouched. tests/fm-watch-triage.test.sh covers the classifier including the .git prune, both halves of the live case on one fixture (quiet plus writing defers, quiet plus silent still escalates and counts), and the bounded re-surface. All three fail without this change. * no-mistakes(review): prove autoarm claims by identity; skip mate-home write probe * no-mistakes(document): document away-mode wedge boundary and probe filesystem limit * no-mistakes(document): qualify turn-end recovery condition for abandoned auto-arm claims * fix(watch): keep a write deferral scoped to its own idle window Two consistency gaps in the worktree write probe, both found while reviewing the wedge-deferral change on this branch. A write deferral is a bounded chain: its .writing-since marker ages the whole chain so a churning worktree still re-surfaces once per resurface window. That is only sound while the chain belongs to the current quiet stretch, so every path that restarts the idle-window timer has to drop it too. Two did not: the corrupt-timer repair in wedge_timer_check, and both first-sight branches for a captain-relevant status. A chain left over from an earlier quiet stretch made the first deferral of the new window re-surface immediately instead of after a full fresh window. FM_WORKTREE_WRITE_PRUNE is a skip list, so clearing it reads as "skip nothing" and is the obvious way to widen the probe to the whole depth-bounded tree. Instead an empty list reported no evidence at all, quietly costing the wedge detector its third liveness input on a home that meant to widen the walk. An empty list now widens the walk, and the header says so. Neither change alters when a stall that writes nothing escalates. Regressions in tests/fm-watch-triage.test.sh cover all three paths and each one fails on the pre-fix code. * no-mistakes(review): honor an empty write-prune, bound the probe, share window_key * no-mistakes(document): align probe knob count and guard regression-coverage ownership * no-mistakes(lint): silence deliberate single-quote SC2016 in write-prune env test * fix(bin): give a captain hold the same bounded pause cadence as a declared pause (kunchenguid#2748) * fix(bin): give a captain hold the same bounded pause cadence as a declared pause Two supervisors read a finished task's last status line and disagreed about which declarations mean an idle endpoint is expected. bin/fm-inactive-reconcile.sh suppresses its inactive-outcome scan only on `captain-held`, while the away-mode daemon's wedge path gated deferral on `paused` alone. Both read the LAST line, so the two verbs are mutually exclusive and no finished task waiting on a person could satisfy both at once. Marking 11 such tasks `captain-held:` silenced the 900s outcome scan and immediately produced five possible-wedge escalations in one batch, because the 240s wedge detector no longer saw a pause verb. fm-classify-lib.sh's status_is_paused_or_captain_held already owns the combined question, and bin/fm-watch.sh's ordinary-crew wedge path already asked it. This extends that same answer to the paths still asking the narrower one: - bin/fm-supervise-daemon.sh, all six sites, which form one subsystem and have to move together. classify_stale returns the pause action, reconcile_pause_tracking and migrate_watcher_pause_markers record and migrate the marker, and housekeeping defers the wedge and then re-surfaces the recheck. Changing only the stale-persistence gate would defer the escalation while reconcile_pause_tracking recorded nothing, so the wedge marker would persist and the sweep would `continue` past it forever: quiet, but never re-surfacing. - bin/fm-watch.sh's secondmate stale gate, whose downstream owner pause_state_class already treats both declarations identically. - bin/fm-push-transition-lib.sh's absorb, where either declaration already names the human the transition would report and the wait is already durably recorded. Quieting alone would be half a fix, so the bounded re-surface had to reach a hold too. A hold has no current-state mapping, unlike `paused`, so authoritative crew state reports it as unknown and pause_state_class received `none`. An ordinary crew recovers pause classification from that state through confirmed agent death, which proves no live decision gate is being silenced. A secondmate's endpoint liveness is deliberately never read there, because an idle mate is healthy by design, so that confirmation is unavailable by construction and cannot be required: without recovering the classification for a mate, every caller silenced a held mate outright and its hold would rot invisibly. That promotion is bounded by the declared-wait guard at the top of the function, so it can only reclassify a task that already declared a wait and shows no positive working evidence. Two narrow `status_is_paused` calls are deliberately left alone. bin/fm-crew-state.sh's map_log_state is a current-state reporting contract, not a wedge path; reporting a hold as `paused` would erase the distinction status_key_closing_verb and fm-captain-hold.sh depend on, where a `captain-held` close is a verified durable transfer and a `resolved` close claims outright settlement. fm-classify-lib.sh's call inside status_is_captain_relevant needs no change because that function's own case list already returns non-relevant for `captain-held`. bin/fm-inactive-reconcile.sh keeps its `captain-held` suppression as it is. Its guard exists because a finished task's crew state still reports done from a higher-priority source than the log, and a declared pause needs no such guard: the scan only reports done or failed, and nothing else reaches its record path. Widening it would change a separate subsystem's reporting contract, which this defect does not require. Coverage extends the existing colocated patterns for these predicates and asserts both halves. tests/fm-daemon.test.sh covers the classification, the wedge marker converting to pause tracking with no escalation, the bounded re-surface with its window reset, and the boundary case where an answered hold stops claiming the cadence. tests/fm-watch-triage.test.sh covers a held secondmate re-surfacing on the same bounded cadence without being labeled a wedge. tests/fm-supervision-events.test.sh covers the absorbed push transition. Every one of these fails on the pre-fix code except the answered-hold boundary case, which is there to pin that the quieting was not widened too far. The `paused:` workaround appended to those 11 tasks is live supervision state and is untouched here. It can be retired once this lands. * no-mistakes(review): name the captain in a held task's bounded recheck * no-mistakes(document): extend declared-wait supervision docs to captain-held holds * fix(bin): make lint prerequisites and harness tests reliable (kunchenguid#2758) * fix(lint): name the installer when ShellCheck or actionlint is missing A missing actionlint exited 127 like a bare command-not-found. Fail with exit 1 and point at the pinned installer, matching the missing-ShellCheck path, without weakening the version pin. * test: isolate kimi and muse detection from inherited Cursor markers Harness detection checks CURSOR_AGENT before ancestry, so these markerless-adapter cases failed when the suite itself ran under Cursor. Clear the verified markers the same way the secondmate harness tests already do. * no-mistakes(document): Document Muse Cursor marker cleanup * feat(bin): report watched tooling updates that are available or installed but inert (kunchenguid#2684) * feat(checks): report tool updates that are available or installed but inert Firstmate had no way to notice that tooling this home depends on needs an update, and no way at all to notice the worse case: an update that installed correctly and then did nothing. That second case is why this exists. A tool that self-installs into ~/.local/bin while a version manager keeps its own older copy earlier on PATH looks completely up to date to anything that asks only "is a newer version published". On 2026-08-20 a Herdr update landed at 0.8.2 while an older 0.8.0 copy stayed earlier on PATH, so every Herdr command failed on a protocol mismatch and firstmate could not read its own fleet. bin/fm-tool-update-check.sh reports the two conditions separately: <tool> update available a newer version exists at the update source. <tool> update not in effect a newer copy is installed on this host, but PATH still resolves an older one. PATH skew is measured, never inferred. Every executable copy of a watched command on PATH is asked for its own version and those answers are compared, so one lookup cannot hide the skew, and a directory name is never read as a version because a version manager's "latest" directory can hold an older build. A copy that will not report a version is a check failure, not a pass. The watched tools live in local, gitignored config/watched-tools.json, so adding a tool is a config edit rather than a code change, and the file is never propagated to another home. Update sources cover both shapes: a local clone's commit distance from its remote branch, and a command's own version and update announcement, including a tool like no-mistakes that prints its version on one command and announces a new release on another. The check prints one line when something needs attention and prints nothing otherwise, so it rides the existing watcher state-check contract with its trust binding instead of introducing a schedule of its own, and state/.tool-updates keeps the same pending update from being reported on every poll. The check only reports. It never installs, updates, reorders PATH, touches a version manager, or fetches into a watched repository; every git probe is read-only. Tests cover the skew case as a regression, and it was verified by mutation: removing the skew report, or stopping after the first PATH hit as a single lookup would, each make that test fail. * no-mistakes(review): fix tool update check probe reporting, budget, and shim write * no-mistakes(review): keep sweeps alive on broken patterns and oversized budgets * no-mistakes(review): roll back failed arm, widen budget clamp, bound repo probe * no-mistakes(review): guard git probes at the budget, record uncut findings * no-mistakes(document): fix stale watched-tool report-record wording in docs and header * no-mistakes: apply CI fixes * no-mistakes: apply CI fixes * no-mistakes: apply CI fixes * no-mistakes: apply CI fixes * no-mistakes: apply CI fixes The behavior shard's watch-triage suite failed on the new worktree-write wedge tests. Those five tests are the only ones in the file that do not use its standard waits. They give a fixed 3 second liveness budget to the one poll that now spawns the bounded worktree walk, and 4 seconds to an escalating watcher where every other test in the file gives 10. On a loaded runner that poll outlives the fixed budget, so the round is reaped before the deferral it asserts on is recorded, and the test reports a lost deferral instead of the deferral under test. Wait for a completed poll cycle through the file's own wait_poll_cycle, which is what its header documents this hazard for, and use the file's standard 100 tick exit budget. Verified against a load that reproduces the failure: 11 of 12 runs failed before, 8 of 8 pass after. Verified by mutation too, so the waits still prove the behavior: removing the write deferral, and keeping a finished deferral chain across an idle-timer repair, each still fail their test. * fix: decouple ask-user decisions from yolo (kunchenguid#2764) * fix: treat yolo as merge authority only, not ask-user finding authority Yolo on/off was documented as also deciding no-mistakes ask-user findings, which hid firstmate's duty to judge unambiguous-toward-design findings itself. Keep every safety boundary; this is a contract clarification, not a relaxation. * no-mistakes(document): Clarify yolo documentation ownership and merge posture * feat(bin): add a spoken interface that answers from records and hands work over (kunchenguid#2767) * feat(voice): spoken round trip on Nova Sonic 2 with a measured relay cost Step one of the spoken interface: the laptop captures and plays audio, this desktop holds the model session, and no AWS credential leaves the desktop. Measured, amazon.nova-2-sonic-v1:0 in eu-north-1, end of speech to first byte of reply audio, 6 runs each, all answered, on a question that forces a records read: relay path 1.229 1.379 1.428 1.447 1.481 1.516 median 1.438 direct 1.147 1.179 1.203 1.237 1.244 1.317 median 1.220 The relay costs about 0.22s of the median. The direct figure reproduces the earlier survey, which is what makes it a usable control. Excluded: the captain's own ssh round trip, microphone capture, and speaker output. This desktop has no microphone and no speaker, so every run used audio files. Three pieces: bin/fm-voice-relay.py holds the conversation on this host bin/fm_voice_records.py what a spoken answer may read, and the handover bin/fm-voice-client.py the laptop end; audio devices UNVERIFIED bin/fm_voice_frame.py the wire format both machines share Real work is handed to the existing bin/fm-inbox.sh rather than a second queueing surface, and the agent says it is handing over rather than answering as firstmate. Read scope: Done history and free-form note bodies are never assembled at any scope, so the wide default cannot reach the places commercial detail accumulates. config/voice-read-scope narrows it to counts only, and config/voice-read-deny excludes a named item in one line. The boundary is an executable test that widening the reader fails. Push to talk is the default because it is cheaper and the choice is still open; --listen open-mic is the single flip. Two traps worth knowing: a clip with no trailing silence is never answered, and the end of a reply is contentEnd with stopReason END_TURN, not completionEnd. A second user turn in one session is treated as barge-in unconditionally, and an interrupted turn that calls a tool is lost, so the session reconnects per turn and gives up conversational memory. That is the concrete thing step three has to solve. * no-mistakes(review): fix voice relay credential reuse, frame validation and record parsing * no-mistakes(review): test uplink header guard, bound unknown expiry, align state dir * no-mistakes(review): decide deny per item, guard turn failures, bound ambient credentials * no-mistakes(review): read account config from home, harden deny and turn failures * no-mistakes(review): close status verb set, fix inbox help, pair data override * no-mistakes(review): keep profile-free relay alive, unblock loop, fix dead assertion * no-mistakes(review): hide finished pull requests, refuse open mic, keep suite offline * no-mistakes(review): survive reader failures, release devices, fix claims A failure while handling a model event, or while sending a tool result, left the reader task dead with ended and turn_done clear, and close() re-raised the stored failure on every await. One dropped stream became a relay that could never build another session. The reader now reports the session over in a finally whatever killed it, and close() absorbs the task the same way it already absorbed its sends. The laptop client releases what it already started when a later startup step refuses, SystemExit from the handshake wait included, and names a device refusal instead of leaking a raw PortAudio error. Whether it releases correctly against a real device is still unverified here. The records docstring claimed every reading was filtered to open ids. Only the pull request count and list are; the worker count and the state histogram cover every live runtime record, finished ids included, because a meta file still on disk still needs tearing down. The finished-work deny half of the suite asserted things that held with the deny list absent. It is replaced by a deny on an open title, which removes the row and says so while the count stays honest. * no-mistakes(review): name reader failures, split file and device refusals A failure inside the model reader released the waiting turn and told nobody. The session was not marked spent, no notice reached the client, and the client waits for a reply end or a notice, so the captain got their whole timeout of silence and then a record saying the turn went unanswered with nothing about why. Both ends of the relay now name a failed turn through one function, once per turn, and --self-test carries the cause in relay_error the way the client's own record does. Two things that are not failures stay that way. A stream that simply ends is the end of a session, which serve still reads on its own terms. A stream that goes away because close() asked it to is an ordinary renew, and announcing it would have put a failure notice in front of the captain on every turn. On the laptop end, the refusal that became a device error covered the file-backed playback and capture too, so a mistyped --in-file was reported as an audio device failure and the advice named the flag that had just failed. The file ends now report the path and the flag that chose it and stay an OSError; the device ends keep the device advice and name the flag for that end. The device paths remain unrun here, so only the file halves are covered by a test. * no-mistakes(test): survive model session end, order client turn frames * no-mistakes(document): sync voice relay docs with reviewed relay behavior * no-mistakes(document): re-measure relay latency and correct its cause * no-mistakes(document): correct measurement date and name the unmeasured SSH hop * no-mistakes(document): describe the unpublished control measurement, fix list formatting * no-mistakes: apply CI fixes * no-mistakes: apply CI fixes * no-mistakes: apply CI fixes * fix(bin): preserve Relay follow-up loops until explicit disposition (kunchenguid#2763) * fix: keep Relay public loops open until retire Delivering a promised-final reply was deleting the only record that tied a public thread to later work, so a follow-on ship silently owed no closing reply. Retain the registration after delivery, rechain follow-on work onto the same thread, and make retire --reason the only close. * no-mistakes(review): Propagate public follow-up registration removal failures * no-mistakes(review): Persist retire receipts and align parent resolution * no-mistakes(review): Make rechain resumable after partial obligation creation * no-mistakes(review): Repair follow-up state, briefs, and expiry escalation * no-mistakes(review): Serialize follow-up delivery stamps with retirement * no-mistakes(review): Serialize rechain claims and protect registration terminal states * no-mistakes(review): Avoid reporting retired delivery loops as open * no-mistakes(document): Refresh public-loop documentation and verification evidence * no-mistakes: apply CI fixes * no-mistakes(review): Preserve delivered follow-up bindings during registration replay * no-mistakes(review): Harden public follow-up retirement and rechain races * no-mistakes(review): Fail closed on unresolved secondmate retirement * no-mistakes(review): Bind secondmate cleanup to its recorded canonical home * no-mistakes(review): Fix rechain command output and expiry validation * no-mistakes(review): Validate brief keys and warn on remote promotion * no-mistakes(document): Document retained public follow-up loops * no-mistakes(lint): Remove unused bounded-wait loop variable * feat(bin): merge GitLab merge requests through the guarded PR merge path (kunchenguid#2779) * feat(bin): merge GitLab merge requests through the guarded PR merge path bin/fm-pr-lib.sh already parses a GitLab merge request URL for the watcher, but bin/fm-pr-merge.sh refused every non-github provider, so a merge request had to be merged by hand and got none of the recording, guards, or audit trail a pull request gets. The merge path now dispatches on the parsed provider. A GitHub URL keeps its exact previous behavior. A GitLab URL is addressed through glab by the project URL rebuilt from the parsed host and path, so a merge request on any instance resolves and no host is hardcoded, and no merge-method flag is added because the project's own merge method is what should apply. A GitLab merge happens only after one live read of the merge request confirms it is open, detailed_merge_status is mergeable, has_conflicts is false, blocking_discussions_resolved is true, and the head pipeline succeeded at the exact current head. Every failing condition is reported, not just the first. The verified head is bound to the merge with glab's --sha, so a push landing between the read and the merge fails the merge instead of landing commits nothing verified. Recorded metadata is never the authority for any of this: a rebase moves the head and leaves a recorded value stale, so a recorded head that disagrees with the live one is reported rather than trusted, and the recorded value is read before the recording step because that step drops a GitLab head it cannot resolve. * no-mistakes(review): reject bundled -R clusters and make tool-absence cases host-independent * no-mistakes(test): state authorised GitHub narrowing of bundled -R guard This branch NARROWS GitHub behaviour. The narrowing was authorised deliberately rather than slipping in by accident, and it applies to both providers, GitHub and GitLab alike, because a script that guards one provider and not the other is a trap for the next reader. What bin/fm-pr-merge.sh now refuses is extra merge arguments containing a bundled short-option cluster that includes R, for example "-dR other/repo". The forge CLIs expand such a cluster one character at a time, so it carries "--repo other/repo", and that later value wins over the repository the URL named. Before this change, "fm-pr-merge.sh <task> <github-url> -- -dR other/repo" reached "gh-axi pr merge 12 --repo example/repo --squash -dR other/repo" and exited 0 with pr= recorded and the merge poll armed. It now exits 1 with "extra merge arguments must not override the repository", records nothing, and invokes no forge merge command. Every other GitHub invocation is byte-identical to the base commit. Closing that hole honours the existing rule rather than departing from it. The file header already forbids --repo and -R because the repository must come only from the URL, so a bundled cluster carrying a repository override was never legitimate behaviour to preserve: it was that guard being evaded. Redirecting a merge to a repository the URL does not name is exactly what the guard exists to prevent. The refusal is already pinned on both paths by the existing case test_bundled_repo_override_args_refuse_before_recording in tests/fm-pr-merge.test.sh. On GitHub ("-dR wrong/repo") and on GitLab ("-yR https://other.example/g/p") it asserts exit 1, the refusal wording, no pr= in the task meta, no armed merge poll, and no forge merge command invoked, with a control case proving a cluster that carries no repository override still reaches the forge. No duplicate assertion was added. Both assertions were confirmed to have teeth by narrowing the guard back to a bare -R and watching each path fail. This commit carries no file change: the guard and its coverage landed in 614853d, and this message exists so the pull request description states the narrowing. * no-mistakes(document): fix README pointer for GitLab watch and merge doc * no-mistakes: apply CI fixes * fix(bin): record a lost relay connection instead of an unanswered turn (kunchenguid#2788) * no-mistakes: apply CI fixes * fix(bin): drop a private record citation and narrow the review rule Three corrections to the spoken interface that landed in kunchenguid#2767, plus one fix carried over from that branch after its pull request had already been merged. The confidentiality fix. The module docstring of bin/fm-voice-relay.py cited a private, gitignored fleet record by exact path and section number. That widens what this public repository points at, and it cannot resolve for any reader here, because the path has never been in the repository. Both traps it pointed at are already described in full in the list immediately below it, and docs/voice-relay.md carries the same two for operators with no citation at all, so the pointer is removed and no claim is weakened by losing it. Two comments that referred to "the survey" as though it were something a reader could open are reworded the same way. Neither exposed a path, so that half is comprehensibility rather than confidentiality. The review rule. .greptile/rules.md is kept, because its conditions are right and deleting it would leave the next reviewer to re-litigate a decision already argued out. What was wrong with it is narrower than its existence: it read as settled repository policy, when whether VISION.md itself should be reconciled is an open question belonging to the captain. One sentence now says so, and says that the conditions listed below it are what the interpretation depends on. That narrows the claim rather than widening it. The carried-over fix. The first commit on this branch is 7f98e79 from fm/voice-relay-build-v4, taken verbatim rather than rewritten. It closes the window where a transport failure was recorded and then erased, so a run could be emitted as answered false with relay_error null. That matters more than it looks: relay_error is the field that keeps an infrastructure failure from being averaged into a latency figure, so the failure mode is a dead connection wearing the costume of a slow reply. It landed fifteen minutes after kunchenguid#2767 merged and so never reached the default branch. * no-mistakes(review): name a reason on every unanswered-turn close path * no-mistakes(review): guard the downlink body and pin frames to their turn * no-mistakes(review): attribute reply audio to its own turn and tell endings apart * no-mistakes(review): tell a cut-short reply from an unanswered turn * no-mistakes(review): discard reply audio arriving after the output closes * no-mistakes(review): count discarded reply audio on the speaker path too * no-mistakes(review): keep a reason off a turn already answered in full * no-mistakes(review): say a reset cut a reply short, not that none arrived * no-mistakes(review): read one turn's audio count once, and hush a tidy exit * no-mistakes(document): fix stale session-end relay_error claim in voice-relay guide * fix(composer): stop a blocked pi pane from proving an empty composer (kunchenguid#2811) A pi worker parked on an interactive prompt - a permission dialog, a question menu, a trust dialog - reports agent_status=blocked, because it is waiting on a human keystroke. Pi draws that menu above its separator pair, so the composer region between the rules is blank and structure alone looks like a free composer. _fm_composer_pi_verdict admitted blocked alongside idle and done, so the shared classifier reported an affirmatively empty composer for exactly the pane where typing is unsafe. Every "is it safe to type here?" consumer reads that verdict and proceeds only on an affirmative empty, so both are told yes on a parked prompt: the away-mode injection guard in bin/fm-supervise-daemon.sh, and fm-send's pre-type refusal. The keys then answer the menu instead of composing a message - the highlighted default is selected, the text is discarded, and the record attributes a decision to a human who never made it. blocked now defers to unknown, which every consumer already treats as fail-closed. idle and done still prove an empty composer, so ordinary steering is unchanged, and Cursor is unaffected because its always-blocked panes never reach this pi-only branch. Regression coverage lands first at both levels: the verdict owner (a blocked pi defers) and the herdr adapter (a parked pi prompt is not an empty composer). * fix(bin): require project clone roots during fleet sync (kunchenguid#2849) * fix(bin): require a clone root before fleet-sync touches a project Git repository discovery walks upward, so `git -C projects/<dir>` on a plain directory nested under projects/ resolves to the enclosing repository - in a firstmate home, the firstmate checkout itself. fm-fleet-sync.sh guarded its candidates with `rev-parse --is-inside-work-tree`, which such a directory passes, so every later git call read, pruned and fast-forwarded firstmate's own default branch and reported it under the project directory's label. A running session's AGENTS.md changed underneath it, and the report named a project that had nothing to do with the change. Require each candidate to be the root of its own work tree before any other git command: compare `rev-parse --show-toplevel` against the directory's own physical path. Both sides are physical, so a symlinked clone still compares equal. Anything else is skipped by name, naming the repository that would have been touched, and bootstrap relays that as a FLEET_SYNC line. Regression coverage reproduces the wrong-repo fast-forward against a home nested inside another repository, in both the whole-fleet and single-project forms, and pins that a symlinked clone dir still syncs. * no-mistakes(review): Keep enclosing fixture clean during clone-root regression * fix(bin): retry transient Lavish poll interruptions (kunchenguid#2846) * fix(procevent): retry a transient Lavish poll interruption quietly A live Lavish listener can be cut short by the server with exactly error: Lavish Editor poll response was interrupted code: SERVER_ERROR while the session's marks remain available. Firstmate registered raw `lavish-axi poll` output, so the generic process-event runner captured that transient response as a result and woke the whole fleet over what is really an internal retry. The Lavish adapter now registers its own listener command, which reruns the published blocking poll up to 12 times at 5 second intervals for that one exact two-line response. The match is deliberately narrow: real feedback, ended and missing sessions, any other SERVER_ERROR, and the same interruption still standing once the bound is spent all pass straight through and are captured and announced as before. The retry is a Lavish fact, so the generic runner stays adapter-agnostic. `FM_LAVISH_POLL_RETRY_DELAY` is a bounded 0 to 60 second override for the interval only, refused rather than rounded when malformed, so a test can exercise the real bound without waiting it out. * no-mistakes(review): Harden Lavish retry matching, validation, and cleanup * no-mistakes(review): Bound Lavish retry staging and stabilize regression * no-mistakes(document): docs: explain Lavish retry adoption * no-mistakes(lint): Restore Lavish trap ShellCheck suppression * fix(brief): stop the documented {TASK} fill from corrupting the Herdr gate (kunchenguid#2838) The unguarded Herdr declaration quoted `{TASK}` in its own prose while the scaffold instructs firstmate to replace every `{TASK}` placeholder. The documented global replace therefore spliced the whole task body into the middle of the safety gate's sentence, silently destroying the one contract that exists precisely because the scaffold cannot inspect the task text. Reword the gate to refer to the task text filled in above, leaving the placeholder only at its genuine fill site. Rewording rather than renaming the token keeps the unfilled-charter guards in fm-home-seed.sh and fm-remote-home-seed.sh working unchanged. Add a regression test that performs the documented global fill on ship and scout scaffolds and asserts the body lands once and the gate survives. * fix(bin): resolve the busy-state lock mtime with the platform's own stat form (kunchenguid#2837) The writer lock's stale-lock branch read the lock's mtime with `stat -f %m ... || stat -c %Y ...`. On GNU coreutils `-f` is filesystem stat, so it consumed the format string as a path, complained on stderr, printed a partial filesystem dump (" File: ...") on stdout, and still exited 0. The GNU form in the fallback therefore never ran, and the following arithmetic evaluated the word `File`, aborting the writer under `set -u` with "File: unbound variable". fm-teardown.sh died there after returning the worktree, leaving state/<id>.meta, .status, .busy-gen, .busy-state, .busy-state.lock/ and .turn-ended behind. The surviving metadata kept the watcher monitoring an endpoint whose agent was gone, so a finished task produced stale wakes forever, and every re-run died identically because the abandoned lock was never broken. Detect the platform once and pick the right stat form, the pattern bin/fm-watch.sh already documents, and treat any non-numeric result as "just created" so a future portability surprise degrades to a lock-timeout refusal rather than killing teardown mid-way. * fix(stow): add opt-in pass horizon for memory decay (kunchenguid#2850) * fix(stow): give memory decay a per-pass horizon so the clock fires The tiered decay clocks were wall-clock only, while admission is per-pass: each /stow admits the findings that pass produced. In a home that stows daily those two rates diverge by the stow cadence, an entry the fleet keeps exercising never reaches 30 days unreinforced, and memory only grows while the pass reports decay evaluated. Give each dated marker an optional unreinforced-pass counter and make both tiers stale at whichever horizon comes first: 10 passes or 30 days for aging, 3 passes or 7 days for perishable. Reinforcement clears the counter and nothing else does, so the existing evidence-based restamp rule stays the only way an entry renews its lease. An absent /N means zero, so entries that stay exercised carry no extra marker bytes, and a rarely stowed home keeps its current behaviour through the unchanged date horizon. * no-mistakes(document): Align stow workflow with dual decay clocks * fix(stow): make the per-pass decay horizon opt-in The unreinforced-pass horizon shipped as a new default archival cadence, which is a product default rather than a restoration of the existing wall-clock contract. Keep the 30-day and 7-day horizons as the only default clock, and put the 10-pass and 3-pass horizons behind an explicit opt-in: config/stow-pass-horizon for the firstmate home, and the file's own header pointer for the public skill. With the opt-in absent no counter is written and no counter is read, so a home that does not ask for it decays exactly as it does today. * no-mistakes(review): Preserve frozen counters and correct archive provenance * test(watcher): stop fixture confirmation budgets racing real child startup (kunchenguid#2876) tests/fm-watcher-lock.test.sh passed in isolation but failed intermittently under full-suite and ambient concurrent load. bin/fm-watch-arm.sh computes its confirmation deadline immediately after forking the real child watcher, so the child's entire fork, exec, lock acquisition and beacon publication has to land inside that wall clock. Two cases shrank that budget to one second, leaving a two-second window for work measured at 3.1-4.9s under CPU oversubscription, so the arm honestly reported "FAILED - no live watcher with a fresh beacon" and their premises collapsed. A third case ran on the production budget, but its child must also execute a registered check before exiting: measured at 1.9-2.3s idle and 9.1-13.1s under load, against an 11s budget. The two cases that must confirm a real child now hold the arm to production's own budget instead of a shrunken fixture one, the immediate-wake case gets an explicit budget with headroom over its measured loaded cost, and the two waits for the arm's typed failure are sized off the largest production default rather than a fixed eight seconds. No bin/ change and no default behavior change: the lock's fail-closed semantics, SIGSTOP handling, stale-heartbeat detection and the arm's typed failures are untouched. Verified 4/4 green at 3x CPU oversubscription (loadavg 75-80) after 3/3 red before the change, and CONTRIBUTING.md records the convention. * fix(bin): deterministically order remote tool paths (kunchenguid#2870) * fix(bin): order discovered tool installs by the shell's own expansion fm_remote_job_compose_operator_path built the asdf and mise install directories with `compgen -G`, which does not sort. Bash sorts glob matches in pathexp.c, on the shell's own pathname-expansion path only; `compgen -G` reaches the same glob_filename through pcomplete.c, which sorts nothing. On bash 3.2 (macOS /bin/bash) and every bash before 5.3 that handed the composition raw readdir order, so which install of a multi-version tool a remote job resolved was decided by directory order on disk rather than by this composition. Expand the globs at the call sites and let the function take the matches, so the composition and the documented portable-PATH contract are the same operation. Quoting the account home at the call site also stops a home whose name contains glob metacharacters from being reinterpreted. The colocated regression pins both the order and the mechanism: bash 5.3 moved sorting into the glob library, so an order-only assertion cannot see the defect there. * no-mistakes(review): Remove source-reading PATH regression guard * no-mistakes(document): fix portable serial shard counts after upstream merge * no-mistakes: apply CI fixes --------- Co-authored-by: Kun Chen <3233006+kunchenguid@users.noreply.github.com> Co-authored-by: Mickaël Rémond <mremond@process-one.net> Co-authored-by: Inthuson <iaminthuson@gmail.com> Co-authored-by: Inthuson <inthuson@amazon.com> Co-authored-by: Christopher McKay <101884182+karotkriss@users.noreply.github.com> Co-authored-by: Daan Aerts <daan@daeverhuur.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Fix the load-sensitive flake in tests/fm-watcher-lock.test.sh (GitHub issue #2844): the suite passes in isolation every time but fails intermittently under full-suite / ambient concurrent load, which taxes every full-suite run and blocks widening test parallelism.
Required approach (investigation first): reproduce under instrumented co-scheduled load and name the exact tripping assertion, then remove the timing sensitivity - event/condition waits over fixed sleeps, or tolerances derived from the component's own configured grace - WITHOUT loosening the watcher lock's fail-closed semantics (SIGSTOP handling and stale-heartbeat detection are safety code). The issue's fallback of quarantining the test from parallel lanes is acceptable only if the sensitivity is irreducible; the PR must state which was chosen and why.
Root cause established by repro (3/3 failures under 2x CPU oversubscription, identical assertion "arm did not start before self-eviction check"): two fixtures set FM_ARM_CONFIRM_TIMEOUT=1, and bin/fm-watch-arm.sh computes its confirmation deadline immediately AFTER forking the real child watcher, so a real fork + exec + lock acquisition + beacon publication had a 2-second wall-clock budget. Measured real startup under that load is 3.1-3.6s, so the arm correctly reported "FAILED - no live watcher with a fresh beacon" and killed the child, destroying the fixture's premise. The production code behaved honestly; the fixture was asking a real process startup to beat a one-second clock.
Re-running the fixed suite at 3x oversubscription surfaced a second case on the same mechanism:
test_arm_propagates_immediate_wake_before_confirmationreturned non-zero after printing the wake it already held. That child must also execute a registered check before exiting, and it runs on the production budget - measured at 1.9-2.3s idle but 9.1-13.1s under load, against 11s. It was a coin flip, not a fixed cost.Fix (test-only, no production change, no default-behavior change):
The suite's own waits stay iteration-counted poll loops, which stretch under load where a wall-clock budget does not. CONTRIBUTING.md records the convention so the pattern does not come back.
Verification: 3/3 red before the change at 2x oversubscription, then 4/4 green at 3x (loadavg 75-80) with the whole suite completing, plus a clean unloaded run. The remaining fixed sleeps in the 40-way lock-contention cases were probed to 6x oversubscription without tripping, so they are left alone rather than rewritten speculatively.
Quarantine was rejected on evidence, not preference: bin/fm-test-isolation-proof.sh already excludes the watcher/wake/lock family, and bin/fm-test-run.sh already refuses --jobs > 1 for this script, yet the reporter's failure happened at --jobs 1 under ambient machine load. Quarantine was already in force and could not have prevented it.
VISION.md alignment: "This repository ships through its own discipline: firstmate work is validated like any other project's, and field incidents become regression coverage" - a flaky red trains contributors to ignore red. "Scripts stop safely and report when the world surprises them" - the arm's refusal was correct, so the test was corrected rather than the safety code weakened. Deliberately no default-behavior change anywhere in bin/.
What Changed
Fixes #2844
Risk Assessment
✅ Low: the test-only change directly addresses the reproduced timing race while preserving production watcher safety behavior and platform-specific confirmation defaults.
Testing
The real-process watcher-lock suite passed normally in 80 seconds and under 2x CPU oversubscription in 225 seconds, including the formerly failing self-eviction startup assertion, real-child peer handoff, immediate wake propagation, typed arm failure, stale-heartbeat detection, and SIGSTOP fail-closed behavior. Reviewer-visible CLI transcripts were captured, production scripts were unchanged, and no transient processes or worktree artifacts remained.
Evidence: Focused watcher-lock test transcript
Source: Focused watcher-lock test transcript
Evidence: Watcher-lock test under 2x CPU oversubscription
Source: Watcher-lock test under 2x CPU oversubscription
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
bin/fm-test-run.sh tests/fm-watcher-lock.test.shStarted 40yesCPU workers across 20 logical CPUs, then ranbin/fm-test-run.sh tests/fm-watcher-lock.test.shVerifiedgit diff --exit-code -- binand confirmed no production script changesVerified all 40 load workers exited and the worktree remained clean✅ **Document** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.