Skip to content

ci: separate trusted policy from repository verification - #16

Merged
openboa merged 11 commits into
mainfrom
codex/bench-admission-contract
Sep 8, 2026
Merged

ci: separate trusted policy from repository verification#16
openboa merged 11 commits into
mainfrom
codex/bench-admission-contract

Conversation

@openboa

@openboa openboa commented Aug 28, 2026

Copy link
Copy Markdown
Member

Delivery status

Merged as 5b4d641 after the owner's explicit initial-landing approval. Codex completed reviewed head 23c840d without findings at 2026-09-08T23:30:28Z and posted a thumbs-up; all twelve review threads are resolved (eleven fixes and one source-backed no-change disposition).

Actual central main CI passed: https://github.com/openboa-ai/.github/actions/runs/34291035106 (job 102277398818). Logs verify the exact merge SHA, tested tree 6146355f63e2b436be685731f698b33ca5b0ddb2, all 28 regressions with zero skips/failures, successful workflow lint, and trusted Gitleaks scanning 38 history commits plus candidate files with no leaks. The named check reports completed/success from GitHub Actions. The review/fix/merge/main-CI monitor is stopped.

No target repository was published or migrated, and no GitHub ruleset, environment or required check was changed or bypassed. Those rollout decisions remain separate. The evidence and initial-rollout discussion below record the preparation and pre-landing gap; they do not mean all Coffee repositories are deployed.

Outcome

Separate shared security and approval policy from repository-owned quality verification across the four Coffee Chat repositories. This supersedes this PR's earlier directory-specific Bench admission patch; it does not create another PR.

Changes

  • Central controls inspect candidate policy as data, without a target bootstrap/parser or Product-layout dispatch.
  • Preserve the trusted required-check name, protected paths, Gitleaks history/worktree/raw-blob scans, dependency review and fail-closed CodeQL.
  • Policy violations and execution failures fail explicitly; they are not converted into approval requests.
  • Run each repository's npm run verify through an immutable-image, non-root isolated launcher. Candidate programs receive no network, host mounts, runner command files, credentials or Docker socket. Dependency installation uses the public hash-locked registry graph with lifecycle scripts disabled.
  • Add central self-CI, canonical PR-only wrapper SHA updates, and organization-owned central main CI.
  • Add read-only live-setting inspection and retain existing team ownership while making independent maintainer review possible.
  • Remove historical Product/Eval dispatchers, including the no-longer-applicable Harbor lane; this does not report calibration or Product performance.

Verified evidence

Candidate commit: 23c840d
Tested tree: 6146355f63e2b436be685731f698b33ca5b0ddb2

  • Central npm run verify: 28 passing regression groups in the actual Linux isolated container; the raw non-UTF8 filename test is Linux-specific.
  • Real isolation probes pass: non-root/capability boundary, network denial, host/credential isolation, workflow-command suppression, failure propagation and cleanup.
  • Current remote main snapshots of all four Coffee Chat repositories pass the new launcher unchanged.
  • Prepared target verifier changes pass: Product 3, Roastery 3, Eval 3, Bench 10 regression groups; all four pass actual isolated execution.
  • actionlint 1.7.12, shell/Node syntax, git diff --check and staged Gitleaks pass.
  • Bench's unpublished 50-case/172-assertion JSON and evaluation material are byte-unchanged by the CI work.

Review fixes

  • Preserve CODEOWNERS file authority and last-match precedence; reject hidden/invalid owners and files GitHub would not load.
  • Query effective main rules with pagination; do not count excluded declarations, missing bypass details, or inert strict-check settings as protection.
  • Correct exact-version matching so routine Dependabot patch/minor bumps remain eligible without losing major/downgrade/range/script/fork boundaries.
  • Parse raw NUL-delimited Git paths and inspect attribute blobs from the exact tree; reject archive omission directives while preserving benign unusual filenames and large attribute files.
  • Lint both workflow extensions, including whitespace/dotfile names, fail closed on empty workflow input, and retain linter failure propagation.
  • Remove candidate-owned pull_request orchestration; keep central PR execution base-owned and report the initial CI evidence gap explicitly.
  • Audit owner-exclusive environment approval, enabled/unknown administrator bypass, and forbidden effective merge queues without changing live settings.
  • Restore the exact PR-only target wrapper contract; reject trigger expansion in policy and classification while preserving canonical immutable SHA updates.
  • Clarify the root security policy's PR-only target contract versus central-only main observation; the exact wording received explicit owner approval.
  • Require root SECURITY.md protection in both base and candidate policies; preserve valid slash forms and owner/Dependabot approval behavior.
  • Investigate the separate-normal-CI finding without speculative changes: normal repository verification is already a mandatory aggregate lane, and current target baselines have no separate status producer. Evidence is in the inline thread.
  • Original findings reproduced before fixes; focused regressions and the full 28-group isolated suite pass. Live CI and latest-head Codex review remain separate gates.

Rollout gate

The owner explicitly approved the exact remaining SECURITY.md correction and initial landing of this existing PR once latest-head Codex re-review has no additional actionable findings, followed by actual central main CI verification. This authorization does not permit changing settings, bypassing required checks, or publishing target repositories. New source code is not proof of live GitHub enforcement.

The candidate-owned supplementary pull_request workflow was removed after review identified that its author could replace host orchestration before any guard/container. Its earlier passing runs are historical candidate evidence, not current-head or trusted CI. The actual base f33da6b has no central CI entry point; latest-head GitHub CI is unavailable, not passing. Complete latest-head Codex review and retain exact-tree local regression/lint/secret-scan/isolation evidence for owner review of the initial landing. Do not fabricate success or bypass required checks. After owner-reviewed landing, observe trusted central main CI before selecting its final SHA for callers. Then align reviewer/ruleset settings and migrate exact target PRs without removing required checks, using only an explicitly approved, scoped and expiring exception where an old pin blocks its own transition. Existing quality/coverage rules remain untouched.

Target implementations are prepared in isolated local commits, not published as Ground Truth and not claimed merged. Product/Judge performance is outside this work.

Tracks openboa-ai/coffee-chat-bench#78.

Preserve required checks, protected paths and owner approval while removing product-layout coupling. Add central self-CI, repository-owned verification isolation, live settings inspection and regression coverage.

Tracks openboa-ai/coffee-chat-bench#78.
@openboa openboa changed the title fix: admit minimal Coffee Chat Bench contract ci: separate trusted policy from repository verification Sep 8, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-08T23:30:28.148263Z 23c840d New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@openboa
openboa requested a review from SonSangjoon September 8, 2026 13:10

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a61d41c9fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/check-candidate-policy.mjs Outdated
Comment thread .github/scripts/audit-ci-settings.mjs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b9e474102

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/classify-candidate.mjs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ae10f86e58

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/run-repository-verify.mjs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 49f70374a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yml Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f18eef7ae2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/pr-verification.yml Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 00d7a0ee32

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/audit-ci-settings.mjs
Comment thread .github/scripts/audit-ci-settings.mjs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b48ca8e35

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/classify-candidate.mjs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 72b878212e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread SECURITY.md Outdated

openboa commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

@codex review

Please review the latest head fc9efaf. The approved SECURITY.md wording correction and evidence are in the existing thread; all earlier scoped code fixes remain included. Please report any remaining actionable issues against this exact head.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc9efaf2ba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/audit-ci-settings.mjs

openboa commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

@codex review

Please re-review unchanged head fc9efaf and reassess the normal-CI finding using the evidence in its inline reply. Repository normal CI already runs in the mandatory quality job (coffee-trusted-gate.yml:234-274 -> run-repository-verify.mjs:68); required depends on quality and check-required-results.mjs:5-6 rejects every non-success quality state. Current remote main merge policies, effective rules and wrapper inventories in all four targets have only the aggregate context, not a separate CI producer. Independent read-only investigation reproduced that quality failure/missing cannot yield a passing aggregate. No code or settings were changed in response to this finding. Please report any remaining concrete actionable issue, including a surviving route if this finding still applies.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc9efaf2ba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/check-candidate-policy.mjs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant