Skip to content
Open
Show file tree
Hide file tree
Changes from 52 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
7728d46
feat(commands): add /extract-audit-knowledge skill (EXP-478)
gvladika May 19, 2026
8d887b1
fix(commands): correct idempotency check in /extract-audit-knowledge …
gvladika May 19, 2026
8460961
feat(audit): seed audit/knowledge/ corpus from existing audit reports…
gvladika May 19, 2026
9acc367
feat(audit-knowledge): backfill 39 missing audits; corpus schema 1.1 …
gvladika May 25, 2026
d51c695
feat(ci): add Stage 1 static-analysis workflow for security review (E…
gvladika May 21, 2026
041e524
chore(ci): remove Aderyn job from Stage 1 workflow (EXP-484)
gvladika Jun 8, 2026
17e1201
feat(ci+commands): add lifi-pr-review skill + Stages 2/3 workflow (EX…
gvladika May 22, 2026
ebec888
fix(ci): scope PR diff to src/ in security-review workflow (EXP-483)
gvladika May 22, 2026
f19e92b
refactor(security-review): wrap ToB skills instead of reimplementing …
gvladika May 22, 2026
0b76783
fix(ci): drop Aderyn refs from Stage 2 workflow + skill (EXP-484)
gvladika Jun 8, 2026
5c08a82
pr-ready: align docs + skip-list with Aderyn removal (lifi-pr-review/…
gvladika Jun 8, 2026
44407dc
docs(security-review): drop EXP-482 noise-baseline references
gvladika Jun 15, 2026
daf85ba
refactor(security-review): drop Aderyn from Stage 1; tighten Slither …
gvladika May 22, 2026
50b0477
fix(semgrep): tighten lf-046 / lf-029 / lf-053 rules (EXP-484)
gvladika May 22, 2026
73316f7
chore(semgrep): delete lf-053-loop-msg-value (duplicate of Slither de…
gvladika Jun 8, 2026
66e40e6
feat(audit-knowledge): coverage check + add-audit chain (EXP-481)
gvladika May 22, 2026
4c4031d
fix(audit-knowledge): drop CI guard, keep local diagnostic (EXP-481)
gvladika May 22, 2026
2fe48e0
fix(audit-knowledge): coverage script honors processed_audits field (…
gvladika Jun 8, 2026
e7de0b7
fix(security-review): tune Slither + Semgrep precision (EXP-484) [sta…
gvladika Jun 15, 2026
2f50581
fix(ci): make Stage 1 actually run (pip slither + own compile, setupt…
gvladika Jun 17, 2026
e92c1cb
Merge branch 'pr/exp-480-static-analysis-workflow' into pr/exp-483-li…
gvladika Jun 17, 2026
52e66e8
feat(ci): Stage 1 is triage input; add id-token for claude-code-action
gvladika Jun 17, 2026
fd83523
pr-ready: pin tob-skills submodule by SHA only, drop branch=main (.gi…
gvladika Jun 17, 2026
4fc13ef
Merge branch 'pr/exp-483-lifi-pr-review-skill' into pr/exp-481-add-au…
gvladika Jun 17, 2026
87c1884
Drop variant-analysis from PR security-review pipeline
gvladika Jun 17, 2026
7b13126
pr-ready: paginate sticky-comment lookup (security-review.yml:319)
gvladika Jun 17, 2026
812707d
Merge branch 'pr/exp-483-lifi-pr-review-skill' into feature/exp-440-s…
gvladika Jun 17, 2026
36ccaa1
pr-ready: relocate OUT_DIR into workspace so curated SARIF uploads (s…
gvladika Jun 17, 2026
d55ef7e
pr-ready: lf-008 exclude admin fns that already emit (lf-008-admin-fn…
gvladika Jun 17, 2026
d75b129
pr-ready: lf-003 drop no-op metavariable-pattern + misleading comment…
gvladika Jun 17, 2026
07349c5
pr-ready: fix draft-PR contradiction in override path (security-revie…
gvladika Jun 17, 2026
bc94591
pr-ready: validate coverage-check inputs, not just JSON syntax (check…
gvladika Jun 17, 2026
e8017b5
Drop empty skip-list/waived files; create waivers on first use
gvladika Jun 17, 2026
d5053bf
Scope Stage 1 SARIF queue to changed files in fp-check triage
gvladika Jun 18, 2026
d2f40c3
Strip ticket IDs, changelog notes, and stale terms from pipeline code
gvladika Jun 19, 2026
9d4a817
Merge branch 'main' into feature/exp-440-security-review-pipeline
gvladika Jun 25, 2026
a6e28ad
chore(security-review): rename skill to lifi-security-review, drop un…
gvladika Jun 25, 2026
be8cdc7
Merge branch 'main' into feature/exp-440-security-review-pipeline
gvladika Jun 25, 2026
103576c
fix(security-review): address code-review findings
gvladika Jun 25, 2026
d093961
fix(security-review): exclude external/fork PRs from keyed Stage 2
gvladika Jun 26, 2026
6779b35
pr-ready: anchor-fix lf-046 receiver name regex (lf-046:43)
gvladika Jun 26, 2026
fdefa35
pr-ready: match ERC20 balanceOf-as-amount in lf-001 (lf-001:32)
gvladika Jun 26, 2026
6491136
pr-ready: narrow lf-029 signature-binding exclusion (lf-029:49)
gvladika Jun 26, 2026
a0428e2
pr-ready: exclude guarded non-native branch in lf-054 (lf-054:24)
gvladika Jun 26, 2026
62a4e42
pr-ready: correct lf-002 SafeERC20 remediation message (lf-002:12)
gvladika Jun 26, 2026
e77a5b0
pr-ready: fix LF-011 title to match root_cause (findings.json:352)
gvladika Jun 26, 2026
da36a79
pr-ready: make extract-audit-knowledge reruns idempotent (extract-aud…
gvladika Jun 26, 2026
78556e7
pr-ready: make Stage 1 SARIF download non-fatal (security-review.yml:…
gvladika Jun 26, 2026
9aa21f7
pr-ready: load security-review controls from base revision (security-…
gvladika Jun 26, 2026
2719783
pr-ready: map src/Interfaces in by-area routing (lifi-security-review…
gvladika Jun 26, 2026
39bc71d
pr-ready: require curated.sarif on cost-guardrail exit (lifi-security…
gvladika Jun 26, 2026
964b777
pr-ready: don't init ToB submodule from PR head (security-review.yml:…
gvladika Jun 26, 2026
e0a0df1
fix(security-review): harden keyed Stage 2 against prompt injection (…
gvladika Jul 2, 2026
6186444
Merge remote-tracking branch 'origin/main' into feature/exp-440-secur…
gvladika Jul 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Custom commands live in `.agents/commands/` (source of truth) and are symlinked

| Command File | Usage | Purpose |
| ----------------- | --------------------------------- | -------------------------------------------------------------------------------------------- |
| `add-audit.md` | `/add-audit` | Add an audit PDF + update `audit/auditLog.json` |
| `add-audit.md` | `/add-audit [--skip-extract]` | Add an audit PDF + update `audit/auditLog.json`, then refresh the security-review knowledge corpus (skip the refresh with `--skip-extract`) |
| `add-network.md` | `/add-network [networkKey]` | Add a new network (networks.json, foundry.toml, permit2Proxy.json, gaszip.json, bridge configs) |
| `add-rule-or-skill.md` | `/add-rule-or-skill` | Standard workflow for adding/updating rules & commands (scoping, dedupe, naming, validation, **skill-authoring principles**) |
| `aikido-address-findings.md` | `/aikido-address-findings [<issue-id> \| all \| pr] [repo-name]` | Aikido triage scoped to the PR (default), a single finding, or the whole repo — ignore false positives, fix real findings in code |
Expand All @@ -71,6 +71,8 @@ Custom commands live in `.agents/commands/` (source of truth) and are symlinked
| `deploy-contract.md` | `/deploy-contract <Contract> <network...> [--production]` | Deploy a facet/periphery to networks + register in each LiFiDiamond (verify, diamondCut/diamondUpdatePeriphery, periphery allowlist). Staging/test terminal path and the deploy primitive `multisig-rollout` calls; production rollouts go through `multisig-rollout` |
| `deprecate-contract.md` | `/deprecate-contract <Name> ...` | Deprecate facet/periphery contracts by removing them from the codebase |
| `deprecate-network.md` | `/deprecate-network <net> ...` | Deprecate networks — remove from networks.json, foundry.toml, deployment logs |
| `extract-audit-knowledge.md` | `/extract-audit-knowledge <audit_id>` | Distill an audit PDF into the `audit/knowledge/` corpus (findings.json + lessons.md + by-area/\*.md) consumed by the security-review pipeline |
| `lifi-security-review.md` | `/lifi-security-review` | PR-time smart-contract security review — LI.FI orchestrator over vendored Trail of Bits skills; emits curated SARIF + sticky-comment summary (Stages 2-3 of `security-review.yml`) |
| `multisig-rollout.md` | `/multisig-rollout <Contract> \| --whitelist-pr <N>` | Orchestrate a production rollout: deploy (delegated to `deploy-contract`) or whitelist-sync across chains, Safe proposals, draft PR, signing hand-off, signature verification, `#dev-sc-multisig-proposals` thread |
| `post-pr-for-review.md` | `/post-pr-for-review` | Post a PR to `#dev-sc-review`, enable auto-merge (squash), tag `@smartcontract_core` |
| `pr-ready.md` | `/pr-ready` | Run CodeRabbit locally against current branch and resolve findings — mandatory final step before opening/updating a PR |
Expand Down
46 changes: 44 additions & 2 deletions .agents/commands/add-audit.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: add-audit
description: Add an audit report to the audit log by parsing a pasted PDF file
usage: /add-audit
description: Add an audit report to the audit log by parsing a pasted PDF file, then refresh the security-review knowledge corpus.
usage: /add-audit [--skip-extract]
---

# Add Audit Report Command
Expand All @@ -16,6 +16,12 @@ This command processes a PDF audit report and automatically:
2. Generates the correct filename according to naming conventions
3. Updates `audit/auditLog.json` with the new audit entry
4. **MANDATORY**: Locates and saves the PDF file to `audit/reports/` with the correct filename
5. **Refreshes the security-review knowledge corpus** by chaining into
`/extract-audit-knowledge` (EXP-481). This keeps `audit/knowledge/findings.json`
+ the markdown projections in lock-step with new audits, so the PR-time
security review (EXP-483) can reference findings from this audit
immediately. Skippable via `--skip-extract` for two-step workflows
(e.g., audit landed but reviewer wants to read the PDF before extraction).

## How to Use

Expand Down Expand Up @@ -155,6 +161,41 @@ When `/add-audit` is invoked with a pasted PDF:
Please verify the commit hash is correct by clicking the link above.
```

9. **Refresh the audit-knowledge corpus** (EXP-481):

This step keeps `audit/knowledge/` in sync with `audit/auditLog.json` so
the security-review pipeline (EXP-483) can reference findings from the
new audit on the next PR run.

- **Default behavior (recommended)**: after the user has verified the
metadata in step 8, invoke `/extract-audit-knowledge <audit_id>` with
the just-generated audit ID. The skill will read the PDF, extract
findings into `audit/knowledge/findings.json`, regenerate
`lessons.md` and the relevant `by-area/*.md` projections, and report
the new findings count.

- **Skip path (`--skip-extract`)**: if the user invoked
`/add-audit --skip-extract`, do NOT chain into extraction. Instead,
print a reminder:

```
Knowledge corpus NOT refreshed (--skip-extract).
Before merging this PR, run /extract-audit-knowledge <audit_id> and
commit the resulting audit/knowledge/ changes — otherwise the new
audit's findings won't be available to the /lifi-security-review
security agent on subsequent PRs.
```

- **Failure handling**: if `/extract-audit-knowledge` errors (e.g., PDF
is corrupted, extraction context exceeded), the audit log entry from
step 7 stays intact. Surface the extraction error to the user and
ask them to either retry, fix the PDF, or proceed with
`--skip-extract` and follow up later.

- **Cost note**: extraction typically uses $0.20–0.50 in Anthropic API
tokens for a single audit. This runs in the developer's local Claude
Code session — no CI billing.

## Validation Checklist

Before finalizing, validate all of the following:
Expand All @@ -169,6 +210,7 @@ Before finalizing, validate all of the following:
- [ ] **Audit ID**: Unique, no duplicates (check existing `audits` section)
- [ ] **Filename**: Follows naming convention, doesn't already exist in `audit/reports/`
- [ ] **PDF file saved**: File exists at `audit/reports/<generated-filename>.pdf` with size > 0 bytes
- [ ] **Knowledge corpus refreshed** (unless `--skip-extract`): `/extract-audit-knowledge <audit_id>` completed and the resulting `audit/knowledge/` files are staged for commit.

## Error Handling

Expand Down
229 changes: 229 additions & 0 deletions .agents/commands/extract-audit-knowledge.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,229 @@
---
name: extract-audit-knowledge
description: Distill an existing audit PDF into the audit/knowledge/ corpus (findings.json + lessons.md + by-area/*.md). Use to seed or update the security-review knowledge base from /audit/reports/.
usage: /extract-audit-knowledge <audit_id>
---

# Extract Audit Knowledge

> **Usage**: `/extract-audit-knowledge <audit_id>` (e.g. `/extract-audit-knowledge audit20250305`).
> If `audit_id` is omitted, ask the user which audit to process.

## Overview

Distills a single audit report PDF into the structured knowledge corpus under `audit/knowledge/`. The corpus is consumed by:

- The PR-time security review skill — `lessons.md` + relevant `by-area/*.md` loaded as cached LLM context
- Future Semgrep rule generation — `findings.json` as the structured rule source
- Human reviewers — markdown files browsable in repo

**v1 scope: single-PDF mode only.** Batch processing of the entire corpus is a manual loop over audits, or a future v2 of this skill.

## Output files (under `audit/knowledge/`)

| File | Source of truth? | Hand-edited? |
| -------------------------------------------------------------------------- | ---------------- | ----------------------------- |
| `findings.json` | Yes (canonical) | Additive only via this skill |
| `lessons.md` | Generated | No |
| `by-area/{facets,periphery,libraries,security,helpers,cross-cutting}.md` | Generated | No |

The two markdown surfaces are projections regenerated from `findings.json` on every run. Never hand-edit them.

## Inputs

- `audit/auditLog.json` — resolves the audit ID's PDF path and metadata
- `audit/reports/<filename>.pdf` — the audit report being distilled
- `audit/knowledge/findings.json` — existing corpus to merge into (if present)

## Schema reference

See section [Finding schema](#finding-schema) at the bottom of this file for the full data model.

## Execution steps

### 1. Resolve target audit

- Parse `audit/auditLog.json`; locate the `audits.<audit_id>` entry
- Extract: `auditReportPath`, `auditedBy`, `auditCompletedOn`, `auditCommitHash`
- Fail with a clear error if `audit_id` not found in the log

### 2. Read PDF in 20-page windows

- Use `Read` with `pages: "1-20"`, then `pages: "21-40"`, … until end of PDF
- Skip front matter (researcher bio, disclaimer, risk-classification tables — typically pages 1-3)
- The findings section is conventionally headed `Findings` or `Issues Found`
- Record page numbers for each extracted finding (the printed footer number, not file-index)

### 3. Apply the skip list

Before extracting any finding, decide whether to keep it. Filtering at this stage prevents corpus bloat that degrades downstream LLM context efficiency.

**Skip (do not extract):**

- Gas optimizations
- Naming, NatSpec, formatting issues
- Code-quality / style / consistency informational items with **no security path** (e.g. "use `isNativeAsset()` instead of `address(0)`")
- Compiler/linter advisories

**Keep:**

- Critical / High / Medium / Low severity (all)
- Informational items with a security-relevant impact (e.g. missing event emissions on admin actions, partial access-control checks)

**When in doubt:** ask "Does this finding describe a way that funds could be lost, frozen, stolen, or operations halted, even hypothetically?" If yes → keep. If no → skip.

This skip criterion is deliberately strict — don't relax it without good reason.

### 4. Extract fields

For each kept finding, extract:

- `title` — single-sentence headline. Rewrite the auditor's title only if unclear or generic
- `severity` — normalized: `critical | high | medium | low | info`
- `severity_native` — auditor's original label (preserves auditor-specific scales)
- `area` — `facets | periphery | libraries | security | helpers | cross-cutting`. Use `cross-cutting` only when the finding touches ≥ 2 areas in non-trivial ways
- `contracts` — Solidity contract names; must match keys in `audit/auditLog.json#auditedContracts`
- `root_cause` — ≤3 sentences, in mechanism terms (why the bug exists), not symptom
- `fix_summary` — ≤2 sentences. Cite commit hash if the PDF includes one (`LI.FI: Fixed in <hash>`)
- `recognition_signal` — single sentence, pattern-ish, useful for matching against new code. Drop the bug specifics; describe the **shape** (see golden sample below)
- `status` — derive from the "LI.FI:" + "Researcher:" lines at end of finding. One of: `raised | acknowledged | fixed | wont_fix | mitigated | reopened`
- `tags` — free-form classifiers (`non-evm`, `eth-transfer`, `reentrancy`, `access-control`, …)
- `source.pages` — exact PDF pages this finding spans

### 5. Assign LF-NNN IDs

- Find the highest existing `LF-NNN` in `audit/knowledge/findings.json` (if the file exists); increment from there
- `source_id` is the composite `<audit_id>::<auditor_label>`, where `auditor_label` is the section number (e.g. `6.1.1`) when available, else the auditor's label (`H-1`, `Issue 14`)
- Prefer section numbers — they are unambiguous within a single audit

### 6. Apply idempotency

Before inserting a finding, compute its `source_id` (`<audit_id>::<auditor_label>`) and search the existing corpus. A finding from this audit run is considered "already processed" if **either** of these matches:

- An existing finding's top-level `source_id` equals the computed `source_id` (i.e., the same audit's first-occurrence is already on file), OR
- An existing finding's `status_history` contains an entry whose reconstructed composite `<audit_id>::<source_label>` equals the computed `source_id` (i.e., this audit's appearance is recorded under a re-audit history of some other finding)

If either matches → **skip insertion** of a new entry. Instead, append a new `status_history` entry to the matching finding (Step 7, "Existing finding").

If neither matches → new entry (Step 7, "New finding").

Re-running the skill on the same audit therefore produces no spurious diffs: each `status_history` row already carries `audit_id` + `source_label`, which reconstruct the exact composite key used for comparison.

### 7. Merge into corpus

For each in-scope finding:

- **New finding** (no `source_id` match anywhere): assign next `LF-NNN`, write a full entry, status_history has one row
- **Existing finding** (some entry's `status_history` already references this `source_id`): append a new `status_history` row to that entry **only if no existing row already carries this `(audit_id, source_label)` pair**; if one does, skip the mutation so a rerun is a true no-op. Do NOT create a new `LF-NNN`. Update top-level `severity` only if the new appearance has higher severity than the recorded one
- **Cross-audit echo** (same root cause appears in two audits under different `source_id`s): do NOT auto-merge. Log a warning to console; let the human decide via follow-up edit. Auto-merging is a future v2 enhancement

### 8. Record processing in `processed_audits`

Append the audit's ID to `findings.json`'s top-level `processed_audits` array (schema 1.1+). This MUST happen even when zero findings were kept — the array records "we considered this audit", distinguishing a zero-finding audit from one that was never processed.

- Read `findings.json#processed_audits` (initialize to `[]` if missing)
- If `audit_id` is not present, append it; keep the array sorted
- If `audit_id` is already present (re-run), no-op (idempotent)

### 9. Regenerate projections

After `findings.json` is updated, regenerate the markdown surfaces:

- **`lessons.md`** — totals tables + flat index of every finding
- **`by-area/<area>.md`** — for each area that has at least one finding, write a file with all entries in that area, sorted severity-descending. Do not write files for areas with zero findings (downstream loaders should handle the missing file gracefully)

### 10. Summary output

Print to console:

- Counts: added / updated / skipped (with a one-line reason for each skipped item)
- Confidence breakdown: high / medium / low
- File diff summary (lines added per output file)
- Spot-check list: 3 random findings for the user to verify by eye

## Validation checklist

Before finalizing the skill's output:

- [ ] All required fields present per schema
- [ ] Every `LF-NNN` ID is unique and monotonically increasing
- [ ] Every `source_id` is unique across the corpus
- [ ] Every finding cites at least one page from the source PDF
- [ ] No skipped categories slipped through (verify a sample)
- [ ] `audit_id` appended to `findings.json#processed_audits` (Step 8) regardless of findings count
- [ ] `lessons.md` totals match actual finding counts
- [ ] `by-area/*.md` files contain only findings tagged with that area
- [ ] `findings.json` is valid JSON (no trailing commas, well-formed strings)
- [ ] Re-running the skill on the same audit produces no diff (idempotency check)

## Finding schema

```jsonc
{
"schema_version": "1.1",
"generated_at": "<ISO-8601 UTC>",
// Schema 1.1+: every audit_id we've considered (with or without findings).
// The coverage check uses this to distinguish "we looked and found nothing"
// from "we never looked". Sorted lexicographically; idempotent on re-run.
"processed_audits": ["audit20240201", "audit20240814", "audit20240902", ...],
"findings": {
"LF-001": {
"id": "LF-001",
"source_id": "audit20250305::6.1.1",
"title": "<single sentence>",
"severity": "critical | high | medium | low | info",
"area": "facets | periphery | libraries | security | helpers | cross-cutting",
"contracts": ["ContractName1", "ContractName2"],
"root_cause": "<≤3 sentences>",
"fix_summary": "<≤2 sentences>",
"recognition_signal": "<single sentence, pattern-shaped>",
"status_history": [
{
"audit_id": "audit20250305",
"source_label": "6.1.1",
"status": "raised | acknowledged | fixed | wont_fix | mitigated | reopened",
"severity_native": "<auditor's original label>",
"notes": "<optional, e.g. 'Verified fix in commit <hash>'>"
}
],
"source": {
"pdf": "<filename relative to audit/reports/>",
"pages": [<int>, ...]
},
"extraction_confidence": "high | medium | low",
"tags": ["<free-form>"]
}
}
}
```

## Golden sample

The canonical reference example is `audit20250305` (Chainflip Facet audit, March 2025):

- 7 raw findings in the PDF → 3 kept (1 high, 2 low) + 4 skipped (1 gas, 3 code-quality informational)
- Demonstrates: all kept-severity tiers, both `facets` and `periphery` areas, `fixed` + `acknowledged` statuses
- Page citations: LF-001 (p.5), LF-002 (p.5-6), LF-003 (p.6-7)

When implementing this skill, validate against the hand-built golden sample before committing.

## Implementation notes

- Use Claude's native PDF reading via the `Read` tool (no external libraries)
- For PDFs >20 pages, batch reads sequentially in 20-page windows
- `source.pages` refers to the printed page number in the PDF footer, not the file-index page
- PDFs sometimes introduce non-breaking spaces or line-wrapping artifacts in code blocks — be lenient when matching extracted text
- The skill performs READ + WRITE operations only on files under `audit/knowledge/` and `audit/auditLog.json`. It must never modify `audit/reports/*.pdf`

## Out of scope for v1

- Batch / `--all` mode that walks all audits in `auditLog.json` (deferred to v2 or done manually)
- Sub-agent fan-out for parallel extraction
- Auto-suggestion of cross-audit echoes via embedding similarity
- Per-field confidence scoring (currently per-finding only)

## Related rules and skills

- `.agents/rules/501-audits.md` — audit log structure and naming conventions
- `.agents/commands/add-audit.md` — the existing `/add-audit` skill that registers new audits in `auditLog.json` (this skill consumes its output)
- `.agents/commands/review-bounty-report.md` — sibling skill for triaging Cantina bug-bounty submissions
Loading
Loading