Skip to content

Expose Codex scan completeness in cost JSON - #2520

Open
NickGuAI wants to merge 3 commits into
steipete:mainfrom
NickGuAI:herd-public-cost-completeness
Open

Expose Codex scan completeness in cost JSON#2520
NickGuAI wants to merge 3 commits into
steipete:mainfrom
NickGuAI:herd-public-cost-completeness

Conversation

@NickGuAI

@NickGuAI NickGuAI commented Jul 30, 2026

Copy link
Copy Markdown

Summary

  • persist whether the requested Codex history window was fully scanned
  • propagate that state through the existing CostUsageTokenSnapshot.historyCoverageIsEstablished field
  • emit historyCoverageIsEstablished in codexbar cost --format json
  • keep unfinished recent-lookback files selected until their bounded parse resumes to completion
  • gate cached completeness on the same native-cache validity checks as cached native totals

Contract

╔══════════════════════════╗
║ Bounded Codex log scan   ║
╚════════════╤═════════════╝
             │ partial/deferred work?
       ┌─────┴─────┐
      yes          no
       │            │
       ▼            ▼
 coverage=false  coverage=true
       └─────┬──────┘
             ▼
 CostUsageTokenSnapshot
             ▼
 cost JSON payload

Consumers can distinguish a cold, partial scan from a complete zero or complete total without parsing diagnostic logs. A successful non-Codex cost snapshot reports true; provider error rows omit the field.

Completeness remains fail-closed while a recently modified session outside the date partition is only partly parsed. A Pi-only or mixed cached result also cannot inherit completeness from a native Codex cache that is invalid for the requested window.

Related consumer work: https://github.com/NickGuAI/monorepo-g/issues/2010

Real CLI behavior

Built and ran CodexBarCLI from this PR's exact head (1b2f75a) on Swift 6.2 / Ubuntu 24.04. Provider-native session files were mounted read-only; the transcript projects only public state and redacts usage values.

First, a cold scan of one completed real Codex session:

$ codexbar cost --provider codex --days 30 --format json --json-only --refresh \
    | jq 'map({provider, hasUsage:(.last30DaysTokens>0), historyCoverageIsEstablished})'
[
  {
    "provider": "codex",
    "hasUsage": true,
    "historyCoverageIsEstablished": true
  }
]

Then, using that cache, the same provider roots were expanded to the real 30-day active and archived corpus. The bounded work exceeded one refresh, and the previously complete state was not reused:

$ codexbar cost --provider codex --days 30 --format json --json-only --refresh \
    | jq 'map({provider, hasUsage:(.last30DaysTokens>0), historyCoverageIsEstablished})'
[
  {
    "provider": "codex",
    "hasUsage": true,
    "historyCoverageIsEstablished": false
  }
]

An independent cold-cache run against the full corpus also emitted false. No prompts, completions, tool content, native log files, private paths, private cache fields, or usage totals are included in this evidence.

Regression coverage

  • an unfinished, recently modified session whose date partition predates the requested window remains incomplete and advances its saved parse offset on the next refresh; only a fully resumed scan reports true
  • Pi-only and mixed snapshots report false when the native Codex cache is invalid for the requested window

Verification

  • make test on Swift 6.2 / Ubuntu 24.04: all 48 discovered selections passed in 4 groups
  • ./Scripts/lint.sh lint-linux portable checks passed on Ubuntu 24.04
  • SwiftLint strict: 0 violations in 1,657 files
  • SwiftFormat lint: 0 changed files
  • generated Codex parser hash check passed (2172c319c3d32054)
  • git diff --check

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Jul 30, 2026
@clawsweeper

clawsweeper Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codex review: needs real behavior proof before merge. Reviewed July 30, 2026, 5:15 PM ET / 21:15 UTC.

ClawSweeper review

What this changes

The branch adds a Codex history-scan completeness field to cost JSON, persists it through the incremental scan cache, and adds --provider-native-only to omit Pi and OMP mirrors from cost scans.

Merge readiness

Blocked until stronger real behavior proof is added - 4 items remain

This PR appears to implement the scan-completeness contract narrowly and covers the two earlier cache-continuation concerns, but it remains merge-blocked by proof and product-contract review: the posted terminal evidence names commit 1b2f75a, while current head df5bcdf adds --provider-native-only and is not demonstrated. Likely related people: iam-brain (medium confidence) for adjacent incremental cost-cache work, and ngutman (medium confidence) for the existing Pi-session merge behavior.

Priority: P2
Reviewed head: df5bcdf7993dcc1d19e03d2f4422272de7165f4b
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The patch is focused and regression-tested, but final-head real behavior proof for the newly added CLI mode is still required before merge.
Proof confidence 🦐 gold shrimp (3/6) Needs stronger real behavior proof before merge: The PR contains useful redacted terminal evidence for the earlier 1b2f75a head, but current df5bcdf adds --provider-native-only and needs a final-head terminal transcript that exercises the changed behavior; redact private paths, usage values, tokens, and credentials. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Needs proof Needs stronger real behavior proof before merge: The PR contains useful redacted terminal evidence for the earlier 1b2f75a head, but current df5bcdf adds --provider-native-only and needs a final-head terminal transcript that exercises the changed behavior; redact private paths, usage values, tokens, and credentials. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed 6 items Current-main integration seam: Current main already accepts includePiSessions in the cost fetcher and merges Pi session reports only when that flag is enabled, so the branch’s CLI flag follows an existing provider-history separation seam rather than adding a parallel scanner.
Coverage propagation and persistence: The branch adds a DailyReportLoadResult carrying scan completeness, persists an optional Codex cache flag, and passes the result through CostUsageFetcher into the CLI JSON payload.
Regression coverage follows prior review concerns: The branch adds scanner coverage for an unfinished recent-lookback file resuming to completion and for Pi-only/mixed cached results when native Codex cache validity does not cover the requested window.
Findings None None.
Security None None.

How this fits together

CodexBar’s cost command scans local Codex and Claude session logs, may merge Pi/OMP mirror sessions, and emits cached token and cost totals for terminal consumers. This PR carries a bounded Codex scan’s completeness state through the scanner and cost snapshot into JSON output.

flowchart LR
    A[Codex session logs] --> B[Bounded history scanner]
    C[Pi and OMP mirrors] --> D[Optional merged history]
    B --> E[Coverage decision]
    D --> F[Cost aggregation]
    E --> G[Cost snapshot]
    F --> G
    G --> H[CLI JSON output]
Loading

Decision needed

Question Recommendation
Should CodexBar commit to historyCoverageIsEstablished and --provider-native-only as supported public cost CLI contracts for downstream automation? Adopt the additive CLI contract: Keep the documented field and opt-in native-only flag, subject to final-head proof of default and native-only behavior.

Why: The implementation is additive, but these names and semantics become compatibility commitments for external JSON consumers and cost-reporting workflows.

Before merge

  • Add real behavior proof - Needs stronger real behavior proof before merge: The PR contains useful redacted terminal evidence for the earlier 1b2f75a head, but current df5bcdf adds --provider-native-only and needs a final-head terminal transcript that exercises the changed behavior; redact private paths, usage values, tokens, and credentials. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • Resolve merge risk (P1) - historyCoverageIsEstablished and --provider-native-only establish new public CLI behavior; maintainers should explicitly accept their semantics for downstream JSON consumers and existing scripts.
  • Resolve merge risk (P1) - The real terminal evidence predates current head and does not show the new native-only path, so the final user-visible contract is not yet proven from the branch that would merge.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Changed surface 11 files; 251 added, 39 removed The patch spans scanner/cache semantics, CLI output and help, focused tests, docs, and one release-workflow guard.
Regression scenarios 2 scanner edge cases added Tests cover deferred recent-lookback continuation and invalid native-cache coverage when Pi data is present.

Merge-risk options

Maintainer options:

  1. Refresh final-head contract proof (recommended)
    Run the current head against redacted real provider data and show both the default merged result and --provider-native-only, including the coverage field where applicable.
  2. Accept the public CLI contract
    A maintainer may accept the additive field and flag as stable downstream behavior after reviewing their documented semantics and upgrade expectations.
  3. Pause the public surface
    If the JSON semantics or native-only mode are not intended as a supported contract, remove or split those additions before merge.

Technical review

Best possible solution:

Adopt the completeness field as an additive, documented Codex JSON contract while preserving merged scans as the default, then merge only after a redacted final-head run demonstrates both default and --provider-native-only output.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR adds an observable completeness contract and an opt-in CLI mode rather than repairing a reported current-main failure. The scanner edge cases are covered by focused tests and a redacted terminal transcript, but the transcript is not from the final head.

Is this the best way to solve the issue?

Unclear: the scanner-to-snapshot propagation is a narrow implementation and addresses the earlier review concerns, but a maintainer must still decide whether the field and native-only flag should be a supported public CLI contract.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 8ef86077e70a.

Labels

Label justifications:

  • P2: This is a bounded CLI and local-history reporting improvement with downstream automation value but no demonstrated urgent regression.
  • merge-risk: 🚨 compatibility: The PR adds a documented JSON field, a persisted cache value, and a new CLI mode that downstream consumers may treat as a stable contract.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR contains useful redacted terminal evidence for the earlier 1b2f75a head, but current df5bcdf adds --provider-native-only and needs a final-head terminal transcript that exercises the changed behavior; redact private paths, usage values, tokens, and credentials. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

What I checked:

  • Current-main integration seam: Current main already accepts includePiSessions in the cost fetcher and merges Pi session reports only when that flag is enabled, so the branch’s CLI flag follows an existing provider-history separation seam rather than adding a parallel scanner. (Sources/CodexBarCore/CostUsageFetcher.swift:2819, 8ef86077e70a)
  • Coverage propagation and persistence: The branch adds a DailyReportLoadResult carrying scan completeness, persists an optional Codex cache flag, and passes the result through CostUsageFetcher into the CLI JSON payload. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:962, df5bcdf7993d)
  • Regression coverage follows prior review concerns: The branch adds scanner coverage for an unfinished recent-lookback file resuming to completion and for Pi-only/mixed cached results when native Codex cache validity does not cover the requested window. (Tests/CodexBarTests/CostUsagePerformanceGateTests.swift:319, df5bcdf7993d)
  • Final-head proof remains incomplete: The PR body’s real terminal transcript says it ran 1b2f75a, but the current head is df5bcdf, whose additional commit introduces --provider-native-only; the supplied transcript does not exercise that final-head behavior. (Sources/CodexBarCLI/CLICostCommand.swift:38, df5bcdf7993d)
  • Repository policy applied: The repository policy asks for focused CLI/parser coverage for CLI-testable behavior and requires make test plus make check before handoff; the branch includes focused CLI and scanner tests, while final-head real CLI proof is still needed. (AGENTS.md:242, 8ef86077e70a)
  • Security and release-workflow pass: The only workflow edit narrows the Homebrew-tap release job to the upstream repository; the diff adds no dependency source, secret access, third-party action, or downloaded executable path. (.github/workflows/release-cli.yml:420, df5bcdf7993d)

Likely related people:

  • iam-brain: Repository release history credits iam-brain for the models.dev pricing/cache pipeline that is adjacent to the incremental Codex scan and cache semantics changed here. (role: adjacent cost-history contributor; confidence: medium; files: Sources/CodexBarCore/CostUsageFetcher.swift, Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift)
  • ngutman: Repository release history credits ngutman for merging supported Pi session usage into Codex and Claude cost history, the behavior that the new native-only CLI mode selectively excludes. (role: adjacent provider-history contributor; confidence: medium; files: Sources/CodexBarCore/CostUsageFetcher.swift, Sources/CodexBarCore/PiSessionCostScanner.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Post a redacted terminal transcript from df5bcdf showing the default result and --provider-native-only result, including historyCoverageIsEstablished where emitted.
  • Update the PR body after adding proof; if automatic re-review does not occur, ask a maintainer to comment @clawsweeper re-review.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (4 earlier review cycles)
  • reviewed 2026-07-30T19:34:06.616Z sha cc4d818 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-30T19:53:18.197Z sha cc4d818 :: needs changes before merge. :: [P1] Keep unfinished lookback files in the coverage decision | [P2] Gate reused coverage on native cache validity
  • reviewed 2026-07-30T20:16:23.026Z sha 1b2f75a :: needs maintainer review before merge. :: none
  • reviewed 2026-07-30T20:44:04.741Z sha df5bcdf :: needs real behavior proof before merge. :: none

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cc4d81844e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +3564 to +3567
$0.codexScanComplete == false
&& $0.touchesCodexScanWindow(
sinceKey: range.scanSinceKey,
untilKey: range.scanUntilKey)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep unfinished lookback files in the coverage check

When a recently modified session outside the requested date partitions is only partially scanned before reaching events in the requested window, its cached days do not touch this range. Subsequent refreshes neither repeat the cold-cache lookback nor select it through cachedCodexSessionFiles, and this predicate consequently ignores its codexScanComplete == false state; with no new per-refresh deferrals, the cache is then persisted as complete even though that file was never finished, so cost JSON can expose historyCoverageIsEstablished: true alongside partial totals. Track unfinished candidates independently of their parsed days or keep the lookback active while prior coverage is false.

Useful? React with 👍 / 👎.

now: now,
historyDays: clampedHistoryDays,
calendar: options.calendar,
historyCoverageIsEstablished: cache.codexHistoryCoverageIsEstablished == true,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate the native cache before reusing its coverage flag

When the Pi cache is valid but the native Codex cache is rejected by the root, timezone, or requested-window checks above, reports can still contain the Pi report and reach this assignment. The coverage value nevertheless comes unconditionally from that rejected native cache; for example, after a complete 30-day native scan and a wider Pi scan, switching to 365 days can hydrate a Pi-only snapshot with historyCoverageIsEstablished: true even though native Codex history for the expanded window was never scanned. Gate this flag on the same native-cache validity checks, otherwise use false when only Pi data is loaded.

Useful? React with 👍 / 👎.

@NickGuAI

Copy link
Copy Markdown
Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 30, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 30, 2026
@NickGuAI

Copy link
Copy Markdown
Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 30, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jul 30, 2026
@NickGuAI

Copy link
Copy Markdown
Author

All contributor-side review findings are resolved, and the downstream deletion-first integration is ready. This PR is the release gate for NickGuAI/monorepo-g#2010, whose accepted contract requires pinning an official CodexBar release rather than a fork commit. When this additive field is acceptable, could it please be merged and included in the next release?

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant