Graphite workflow attribution suite: 540 empirical scenarios + failure taxonomy - #2021
Open
scouredimage wants to merge 6 commits into
Open
Graphite workflow attribution suite: 540 empirical scenarios + failure taxonomy#2021scouredimage wants to merge 6 commits into
scouredimage wants to merge 6 commits into
Conversation
scouredimage
marked this pull request as ready for review
July 29, 2026 16:45
Empirically-grounded scenario matrix for proving AI attribution survives every Graphite workflow permutation: workflow catalog from ordered debug-log traces (gt sync's 107-command anatomy incl. reset -q --keep for checked-out moves and the synthetic-base commit-tree/merge-tree rewrite core), fleet command-shape summaries from 7 engineers, gt binary string mining, and the restack-undo repro. generate_matrix.py is the single source of truth and emits scenarios.json for the Rust harness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…im, assertions) Harness core for the Graphite workflow attribution suite (PLAN.md): - scenario.rs: serde loader for the 540-scenario scenarios.json manifest with typed dimension enums (StackShape/Attribution/TrunkState/ Observation/Repeat) and family/by_id filters. - stackbuilder.rs: builds the five stack shapes with 100%-AI files (mock_ai checkpoints + traced commits), optional bare origin, and the trunk ff/diverged/overlap fuel (raw, trace2-off); returns StackState with per-branch expected lines + note commits. - gt_sim.rs: faithful replays of gt's command sequences — synthetic-base restack (cat-file/commit-tree -m _/merge-tree/commit-tree/update-ref or update-ref --stdin batch), reset -q --keep checked-out moves, stash create snapshots, conflict fallback to real rebase with continue/abort, undo, create/modify/submit, move --onto, housekeeping, 1.7.x hash-object storms, linked-worktree execution, partial staging, and lifecycle composites; traced vs blind Exec layer with blind recovery (daemon restart + traced poke). - assertions.rs: non-panicking invariant checker (all AI-file lines blame to AI; every surviving AI commit has a note with attestations), returning per-violation descriptions keyed by scenario ID. - smoke.rs: GT-SYNC_RESTACK-001 end-to-end through the pipeline (passes). - test_repo.rs: git_with_stdin (traced twin of git_with_env) and git_og_with_stdin_and_env (raw twin) for update-ref --stdin / hash-object -w --stdin flows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cenarios) One #[test] per (family x observation x repeat) bucket, each iterating its scenario slice through gt_sim::run_scenario and reporting all violations in a single panic keyed by stable scenario IDs. Generated mechanically from scenarios.json; regenerate via the tooling rather than editing by hand. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Throughput: the traced Exec layer no longer runs sync_daemon_force after every tracked command — traced traffic rides the async trace2 pipeline (closer to production). Barriers are kept only where semantics demand a processed daemon: end of each workflow round (between repeat rounds), before the blind-recovery daemon restart, before a linked worktree is removed (per-worktree reflogs), and at the start of the assertion phase (git-ai blame / git notes reads also pre-sync on their own). Robustness: run_scenario now executes each scenario on a watchdog thread with a 180s wall-clock deadline; on expiry it returns a distinguishable "TIMEOUT" violation and abandons the thread, so a hung daemon costs one scenario instead of the bucket. Scenario panics still propagate. The once-per-process git-ai binary build is warmed before the clock starts, and each scenario logs a "[gt-sim] <id> finished in Xs" timing line. GT-SYNC_RESTACK-001 scenario time: 13.4s (was ~68s wall for a solo run, which was dominated by the pre-existing once-per-process cargo build of the git-ai test binary; marginal scenarios in a bucket now run 7-15s). SYNC_FF traced/once bucket (8 scenarios): all pass, 7.5-14.5s each. Also: rustfmt normalization of the generated family bucket files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…merges The restack core assumed single-commit branches: replay_branch_commit anchored the synthetic base at old_tip~, which for MERGE_IN_STACK's two-commit merge-carrying branch is the branch's OWN AI commit — the 3-way merge then resolved to exactly the new trunk tree and the branch's content (b1_ai.txt) was erased from the rewritten history. restack_range now replays the branch's own commits individually: - enumerate them via rev-list --reverse --first-parent <old_parent>..<old_tip> and replay bottom-up, threading the new parent and recording per-commit old->new mappings; - non-merge commits keep the existing synthetic-base replay shape; - merge commits mint REAL merges: merged tree via the synthetic-base trick against the new first parent, then commit-tree with the new first parent plus every non-first parent (kept as-is when unchanged, remapped when replayed); - per-commit skip when a commit already sits on the right parent (subsumes the old whole-branch idempotency check); - branch.ai_commit_sha is retargeted through the per-commit map (A1 -> A1'), falling back to the tip only for the mapless rebase path. MERGE_IN_STACK traced/once: both scenarios now pass (0 violations). MERGE_IN_STACK blind/once: still fails via the blind-window note strand (content survives, notes never migrate to the raw-rewritten commits) — a genuine finding, left failing for triage. Smoke re-verified green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
540 scenarios: 332 pass, 208 fail, all failures triaged to root cause. Genuine application gaps: (1) unobserved non-FF ref moves strand notes with no reconciliation path (gt's plumbing writes no rebase spans, so even span-keyed recovery cannot fire); (2) pending working-log loss via sideways reset --keep onto minted siblings (src/daemon.rs:5648, forward- only carry), cross-worktree update-ref (src/daemon.rs:5734), and stash pop -q flag mis-parse (src/daemon/ref_cursor.rs:922). Validated healthy: forward resets (git-ai-project#1976), undo bounds (git-ai-project#1978/git-ai-project#1983), sync-FF, traced amends, conflict-abort. One harness bug found and fixed (multi-commit merge replay). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scouredimage
force-pushed
the
graphite-workflow-attribution-suite
branch
from
July 29, 2026 16:50
0c192d4 to
09c569e
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
An exhaustive, empirically-grounded scenario suite proving where AI attribution survives — and precisely where it does not — under Graphite's real command sequences. Graphite drives most attribution incidents at our deployment, and its workflows never run
git rebase: restacks arestash createsnapshots + synthetic-basecommit-tree→merge-tree --allow-unrelated-histories→commit-tree(real msg)→update-ref(orreset -q --keepfor the checked-out branch). The suite replays those exact sequences, captured from ordered gt debug-log traces on one machine plus redacted command-shape summaries from six more engineers (1,971 gt invocations, gt 1.7.13–1.8.6) and string-mining of the gt binary.Layout
tests/integration/graphite_workflows/PLAN.md— workflow catalog (with confidence tags), harness design, triage protocolgenerate_matrix.py→scenarios.json— 540 scenarios / 19 families (stack shape × attribution state × trunk divergence × daemon observation × idempotency, plus worktrees,update-ref --stdin, renames, in-stack merges, partial staging, lifecycle chains, nested conflicts); stable append-only IDsgt_sim.rs/stackbuilder.rs/assertions.rs— replay layer, 100%-AI content construction (mock_ai), non-panicking exact line-blame + note assertions#[test]s (one per family × observation × repeat), each reporting every violation keyed by scenario IDEXPECTED_FAILURES.md— the first full run's complete triageFirst full run: 332 pass / 208 fail — every failure root-caused
Confirmed healthy under adversarial permutations (all pass): forward
reset --keepworking-log carry (#1976), restack-undo bounds (#1978/#1983), sync fast-forward, traced amend cascades, conflict-abort round-trips, 1.7.x object-storm flavors.Genuine gaps found (buckets left failing deliberately — they document real bugs; happy to convert to
#[ignore = "known: …"]if you'd rather gate):reset: moving to/update-ref reflog rows — no rebase spans — so nothing reconciles after restart; notes stay stranded on unreachable pre-restack commits. Note this also means Reconcile rebases that completed while the daemon was not observing #1961 as currently scoped (rebase-span keyed) would not fire here; the fix direction is reconciling arbitrary unobserved non-FF branch moves whose old tip is noted and new tip is not.a. sideways
reset -q --keeponto a commit-tree-minted sibling doesn't carry the working log (src/daemon.rs:5648-5673renames only for ancestor moves; Fix working log getting lost onreset --keep#1976 is forward-only) — 4-command minimal repro in EXPECTED_FAILURES.md;b.
update-refmoving a branch checked out in a different worktree doesn't carry it either (src/daemon.rs:5713-5766only consults the invoking command's HEAD);c.
stash pop -qnever restores stash-saved attribution —enrich_stash(src/daemon/ref_cursor.rs:916-931) takes-qas the stash target; existing stash tests are all flag-less.base_commit_sha= itself, sessions present, no entries).One harness bug was found by the same triage and fixed in-suite (multi-commit branches with merges were being erased by the replay core; merges are now minted as real merges with remapped parents).
Running
cargo test --test integration graphite_workflows::(≈10-15s/scenario, 180s per-scenario watchdog; full matrix ~2-3h at 4 threads). Single bucket: filter by name; single scenario: seesmoke.rs.Related: #1961 #1975 #1976 #1978 #1983.
🤖 Generated with Claude Code