Skip to content

Add automated Wasp code review - #4554

Closed
infomiho wants to merge 21 commits into
mainfrom
miho/wasp-review-skill
Closed

infomiho wants to merge 21 commits into
mainfrom
miho/wasp-review-skill

Conversation

@infomiho

@infomiho infomiho commented Jul 27, 2026 •

Copy link
Copy Markdown
Member

Description

Adds automated PR review.

The workflow runs the /wasp-review skill with gpt-5.6-sol against trusted same-repository PRs in a read-only Codex job, then a separate trusted publisher adds up to five new inline findings, keeps or resolves existing review threads, and updates one summary comment. Draft and fork PRs are skipped.

Example run: #4620

Alternatives

CodeRabbit is a reasonable replacement if we no longer want to maintain this workflow. It is free for public open-source repositories. Downside: it can read our review skill as instructions, but it cannot run the skill directly or let us choose the exact model.

pr-agent a PR review system that is also bring-your-own-key. It was the free offering of a Qodo and now it's donated to community, might be abandoned later?

kodus-ai they are a self-hosted / open-source CodeRabbit alternative. Could be all we need, but we need to maintain a service (maybe we use our Coolify instance for this and call it a day?)

Type of change

  • 🔧 Just code/docs improvement
  • 🐞 Bug fix
  • 🚀 New/improved feature
  • 💥 Breaking change

Checklist

  • I tested my change in a Wasp app to verify that it works as intended.

  • 🧪 Tests and apps:

    • I added unit tests for my change.
    • (if you fixed a bug) I added a regression test for the bug I fixed.
    • (if you added/updated a feature) I added/updated e2e tests in examples/kitchen-sink/e2e-tests.
    • (if you added/updated a feature) I updated the starter templates in waspc/data/Cli/templates, as needed.
    • (if you added/updated a feature) I updated the example apps in examples/, as needed.
      • (if you updated examples/tutorials) I updated the tutorial in the docs (and vice versa).
  • 📜 Documentation:

    • (if you added/updated a feature) I added/updated the documentation in web/docs/.
  • 🆕 Changelog: (if change is more than just code/docs improvement)

    • I updated waspc/ChangeLog.md with a user-friendly description of the change.
    • (if you did a breaking change) I added a step to the current migration guide in web/docs/migration-guides/.
    • I bumped the version in waspc/waspc.cabal to reflect the changes I introduced.

No Wasp app, e2e, starter, documentation, changelog, or version changes are needed for this repository automation.

@infomiho infomiho changed the title Add project code review skill Add Wasp code review skill Jul 27, 2026
@infomiho
infomiho temporarily deployed to railway-deploy-test July 27, 2026 10:25 — with GitHub Actions Inactive
@infomiho
infomiho temporarily deployed to fly-deploy-test July 27, 2026 10:25 — with GitHub Actions Inactive
@pkg-pr-new

pkg-pr-new Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

@wasp.sh/spec

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/spec@4554

@wasp.sh/wasp-cli

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli@4554

@wasp.sh/wasp-cli-darwin-arm64-unknown

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-darwin-arm64-unknown@4554

@wasp.sh/wasp-cli-darwin-x64-unknown

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-darwin-x64-unknown@4554

@wasp.sh/wasp-cli-linux-arm64-glibc

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-linux-arm64-glibc@4554

@wasp.sh/wasp-cli-linux-x64-glibc

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-linux-x64-glibc@4554

@wasp.sh/wasp-cli-linux-x64-musl

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-linux-x64-musl@4554

commit: e7671c4

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

Deploying wasp-docs-on-main with  Cloudflare Pages  Cloudflare Pages

Latest commit: e7671c4
Status: ✅  Deploy successful!
Preview URL: https://cab3ed9e.wasp-docs-on-main.pages.dev
Branch Preview URL: https://miho-wasp-review-skill.wasp-docs-on-main.pages.dev

View logs

@infomiho
infomiho temporarily deployed to fly-deploy-test July 27, 2026 10:45 — with GitHub Actions Inactive
@infomiho
infomiho temporarily deployed to railway-deploy-test July 27, 2026 10:45 — with GitHub Actions Inactive
@infomiho
infomiho marked this pull request as ready for review July 27, 2026 12:59
@infomiho
infomiho requested a review from a team as a code owner July 27, 2026 12:59
@infomiho
infomiho requested review from sodic and removed request for a team July 27, 2026 12:59
@infomiho

Copy link
Copy Markdown
Member Author

Testing the review action by marking the PR ready for review.

@github-actions

github-actions Bot commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Code review

One publishing reliability issue remains in the review schema contract.

@infomiho
infomiho temporarily deployed to railway-deploy-test July 27, 2026 13:07 — with GitHub Actions Inactive
@infomiho
infomiho temporarily deployed to fly-deploy-test July 27, 2026 13:07 — with GitHub Actions Inactive
@infomiho
infomiho temporarily deployed to railway-deploy-test July 27, 2026 13:40 — with GitHub Actions Inactive
@infomiho
infomiho temporarily deployed to fly-deploy-test July 27, 2026 13:40 — with GitHub Actions Inactive
@infomiho infomiho changed the title Add Wasp code review skill Add automated Wasp code review Jul 27, 2026
@infomiho
infomiho changed the base branch from main to miho/wasp-review-skill-only July 27, 2026 13:44
@infomiho infomiho mentioned this pull request Jul 27, 2026
1 of 15 tasks

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not part of the existing ci workflow?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep it separate because ready_for_review would need to retrigger the entire CI workflow.

Comment on lines +20 to +23
contains(
fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'),
github.event.pull_request.author_association
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the "label PRs as external", I tried doing stuff with these tags but they don't straightforwardly signal what I think we'd like them to. I ended up using only the github.event.pull_request.head.repo.full_name == github.repository check, and I think here you can do it too.

review: ${{ steps.review.outputs.final-message }}
steps:
- name: Checkout pull request
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use sha-pinned actions anywhere else. (Yes they are good for security but they need an autoupdater too so we don't miss out on important improvements)

- name: Checkout pull request
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: refs/pull/${{ github.event.pull_request.number }}/merge

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm quite sure this is already the ref when using the checkout action inside a PR trigger, i think

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the default for pull_request, but I'm keeping it explicit because this is also the pattern in OpenAI's official Codex review example: https://github.com/openai/codex-action/blob/dd78cb653811af44014baa08fe954e28d32c1bf9/README.md#L35-L46

ref: refs/pull/${{ github.event.pull_request.number }}/merge
persist-credentials: false

- name: Fetch pull request refs

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this for?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's to bound the agent to review base and head refs, why don't we use the same SHAs here as you do in the message to the agent?

publish:
name: Publish review
needs: review
if: ${{ needs.review.result == 'success' }}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if: ${{ needs.review.result == 'success' }}

This is already the default

Comment on lines +119 to +121
with:
github-token: ${{ github.token }}
script: |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're looking for this instead gh pr comment --edit-last --create-if-none ...

env:
REVIEW_JSON: ${{ needs.review.outputs.review }}
run: |
printf '%s' "$REVIEW_JSON" | node .github/wasp-review/to-rdjson.mjs \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a meaningful difference?

Suggested change
printf '%s' "$REVIEW_JSON" | node .github/wasp-review/to-rdjson.mjs \
echo "$REVIEW_JSON" | node .github/wasp-review/to-rdjson.mjs \

Comment thread .github/wasp-review/to-rdjson.mjs Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this might be much simpler if we define the schema in zod and then write it to this file with z.toJsonSchema(), so we can use the same values and types everywhere

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented .github/code-review package with Zod schemas, thanks for the suggestion!

Comment thread .github/wasp-review/to-rdjson.mjs Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're lucky i didn't find any process.exit() here 🫡

@infomiho
infomiho temporarily deployed to railway-deploy-test August 4, 2026 10:08 — with GitHub Actions Inactive
@infomiho
infomiho temporarily deployed to fly-deploy-test August 4, 2026 10:08 — with GitHub Actions Inactive
Base automatically changed from miho/wasp-review-skill-only to main August 4, 2026 10:23
@infomiho
infomiho force-pushed the miho/wasp-review-skill branch from b65ea4d to bf26932 Compare August 4, 2026 10:35
@infomiho
infomiho temporarily deployed to railway-deploy-test August 4, 2026 10:46 — with GitHub Actions Inactive
@infomiho
infomiho temporarily deployed to railway-deploy-test August 5, 2026 11:46 — with GitHub Actions Inactive
@infomiho
infomiho force-pushed the miho/wasp-review-skill branch from 30d4fb7 to e7671c4 Compare August 5, 2026 17:31

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated code review findings.

GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REPOSITORY: ${{ github.repository }}
run: npm --prefix .github/code-review run publish-review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a same-repository PR changes .github/code-review/package.json or commands/publish-review.ts, this step executes that PR-controlled code with pull-requests: write. A contributor can then post or resolve review content directly instead of being limited to the validated Codex result. Run the publisher from a trusted base-branch checkout or a pinned action, and keep the PR checkout only as the data being reviewed.

@infomiho
infomiho temporarily deployed to railway-deploy-test August 5, 2026 17:59 — with GitHub Actions Inactive
@infomiho
infomiho temporarily deployed to fly-deploy-test August 5, 2026 17:59 — with GitHub Actions Inactive
@infomiho

infomiho commented Aug 6, 2026

Copy link
Copy Markdown
Member Author

Ok, now we have Pullfrog enabled that basically does this for us:

  • runs reviews (automatically or we can set it up only when mentioned)
  • uses GPT 5.6 Sol and our review skill

That's basically what we wanted and I don't think we should write our own action we maintain anymore. Let's see if we like Pullfrog first?

cc: @cprecioso

@cprecioso

Copy link
Copy Markdown
Member

Let's see if we like Pullfrog first?

sounds good, unsubscribing from this PR then

@infomiho

infomiho commented Aug 6, 2026

Copy link
Copy Markdown
Member Author

Closing in favor of Pullfrog, let's see how we like it.

@infomiho infomiho closed this Aug 6, 2026

This branch was previously deployed

2 inactive deployments
fly-deploy-test — e7671c40 Deployed Aug 5, 2026 by infomiho via deploy-test / Test Fly deployment #11478
railway-deploy-test — e7671c40 Deployed Aug 5, 2026 by infomiho via deploy-test / Test Railway deployment #11478
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants