Skip to content

ci: test the effect peer range, Bun and workerd nightly and before release - #1244

Merged
Mkassabov merged 8 commits into
mainfrom
ci/compat-nightly
Nov 15, 2026
Merged

Mkassabov merged 8 commits into
mainfrom
ci/compat-nightly

Conversation

@Mkassabov

@Mkassabov Mkassabov commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1234.

CI tests one setup: Node 24 with the lockfile's effect. The packages promise the whole effect peer range, the Node versions effect supports (engines: node >=22.19.0), and bun, worker and workerd export conditions. This adds a nightly run of those, which keeps one issue while it fails, and runs the same tests before every release. No AI is involved: each job passes or fails on its own checks, and the issue comes from scripts/compat/report-issue.ts with the gh CLI.

What runs (compat-tests.yml, a reusable workflow; each job takes 20-40 s)

  • Effect (lowest / latest in the peer range): scripts/compat/effect-version.ts reads the catalog range, picks the floor or the newest stable release older than the workspace's 1-day minimum release age, and adds overrides for effect and @effect/platform-node. Then pnpm install --no-frozen-lockfile and pnpm vitest run.
  • Node 22.19.0, 22, 26: pnpm vitest run on effect's lowest supported Node, the newest 22.x and the newest LTS. CI already runs 24.
  • Bun (core tests): scripts/compat/bun-tests.ts runs core's tests with bun --bun (Bun 1.4.4, pinned) and compares the failures with scripts/compat/bun-failures.txt, the way the protobuf conformance runner uses protobuf-failures.txt: a new failure fails the job, and so does a listed test that passes.
  • workerd: benches/bundlers/workerd/workerd.test.ts packs the SDKs as the Bundlers job does, bundles the shared calls with wrangler's settings, and runs the bundle with workerd test (workerd 1.20261026.1, already in the lockfile through alchemy). The Worker must print the same lines as the unbundled program in Node. Before this, the Bundlers job built the workerd bundle and ran it in Node.

When it runs

  • ci.yml, every PR, push and merge-queue run: every job except Effect latest, as the Compatibility / ... checks. A new effect release would fail every open PR, so the newest effect runs only nightly and before a release.
  • compat.yml: nightly at 04:17 UTC (all jobs), and by hand. A scheduled run keeps one issue, "Nightly compatibility run failed":
    • a failure opens it, or reopens it if it is closed (also when someone closed it while the run still failed);
    • a failure with the issue open updates its body (failing jobs, first and last failed run) and comments only when the set of failing jobs changes;
    • a passing run comments and closes it.
  • release.yml: a new compat job with all jobs; build needs it, so a failure stops the release before anything is committed or published.
  • stacks/distilled-monorepo: the 6 Compatibility / ... checks join the merge queue's required checks. Deploy after this merges: env -u CI pnpm exec alchemy deploy --stage prod --profile alchemy-admin --include merge-queue.

Other changes

  • packages/core/test/protocol-grpc.test.ts: afterAll destroys the http2 server sessions first. On Node 22, server.close() waits for the client's idle sessions, so the hook timed out.

Known failures today

  • Bun 1.4.4: one listed failure. new Blob([..], { type: "text/html" }) reports text/html;charset=utf-8, and core sends the Blob's own type.

Checked

  • A temporary pull_request trigger (reverted) ran the workflow on this PR: every job passes except Effect latest, the known failure above.
  • report-issue.ts against a stub gh: create, quiet updates, 1 comment on a changed failure set, reopen after a manual close, close on pass, reopen on the next failure; one issue throughout.
  • All 6 Compatibility / ... checks pass on this PR's CI.
  • workerd test exits 1 when a line differs.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Benchmarks: PR 9d73b41be vs main eb62291aa

  • 🔴 Bundle size 54.7 KB: +1.2% (+657 B), bigger than main
  • ⚪ Runtime speed 2.6 µs: +0.2% (+0.0 µs), same as main
  • ⚪ Runtime memory 794.5 KB: +0.0% (+160 B), same as main
  • ⚪ Objects at import 1,082: ±0.0% (±0), same as main
  • ⚪ Whole-SDK memory aws 39.7 MB (−0.0%), cloudflare 8.27 MB (−0.0%), gcp 24.8 MB (+0.1%)
  • ⚪ Repo size 538.2 MB tracked in 9,693 files, same as main (+21.1 KB). .generated-specs/ 0 B, src/services/ 508.9 MB, everything else 29.4 MB.

sdk-memory

bundle-programs

memory-programs

bundle memory
speed objects

Numbers
Metric Change vs main Programs
Bundle (gzip) 17 worse 17
Import heap no change 6
Objects at import no change 6
Added latency (p50) ¹ no change 6
Import time ¹ no change 6
First call ¹ no change 6
Program Call Bundle (gzip) Import heap Objects at import Added latency (p50) Import time First call
aws s3.listBuckets 74.2 KB (+0.7%) 🔴 — — — — —
aws~awsJson1_0 dynamodb.listTables 67.1 KB (+0.8%) 🔴 — — — — —
aws~awsJson1_1 kms.listKeys 66.4 KB (+0.8%) 🔴 — — — — —
aws~awsQuery sts.getCallerIdentity 67.8 KB (+0.8%) 🔴 — — — — —
aws~ec2Query ec2.describeRegions 67.8 KB (+0.9%) 🔴 — — — — —
aws~restJson1 lambda.listFunctions 68.4 KB (+0.8%) 🔴 — — — — —
azure resources.ListTenants 57.0 KB (+1.1%) 🔴 — — — — —
cloudflare workers.listScripts 69.2 KB (+1.1%) 🔴 — — — — —
cloudflare~CloudflareProtocol ips.listIps 49.8 KB (+1.3%) 🔴 — — — — —
fly-io machines.getRegions 50.8 KB (+1.2%) 🔴 775.9 KB 1,088 3.7 µs 8.8 ms 3.4 ms
fly-io~FlyApiProtocol mpg.listUsers 50.6 KB (+1.3%) 🔴 667.9 KB 982 2.4 µs 8.3 ms 3.2 ms
fly-io~SpritesProtocol sprites.createOrganizationToken 51.2 KB (+1.2%) 🔴 706.1 KB 1,024 5.8 µs 8.5 ms 3.6 ms
gcp storage_v1.listBuckets 50.6 KB (+1.2%) 🔴 — — — — —
github gists.list 53.4 KB (+1.2%) 🔴 656.5 KB 954 2.6 µs 7.4 ms 3.4 ms
neon neon.listProjects 52.1 KB (+1.2%) 🔴 836.5 KB 1,082 1.7 µs 8.2 ms 3.4 ms
resend resend.listLogs 51.8 KB (+1.2%) 🔴 859.2 KB 1,144 1.6 µs 8.4 ms 3.2 ms
stripe stripe.GetFiles 50.1 KB (+1.2%) 🔴 — — — — —

PR 9d73b41be against main eb62291aa (main's cached run from 2026-11-15 18:59 UTC). 17 programs, measured in 65 s on 8 vCPU.

Package Tarball Unpacked Files
core 325.9 KB (+0.1%) 1.3 MB (+0.1%) 229
fly-io 249.7 KB 1.6 MB 71
github 2.1 MB 19.6 MB 338
neon 218.5 KB 1.2 MB 47
resend 173.2 KB 1.2 MB 57
Tracked files Size Change vs main Files
services 508.9 MB ±0 B 2,969
in-tree-specs 98.3 KB ±0 B 4
patches 10.0 MB ±0 B 4,145
handwritten 7.1 MB +380 B 1,875
benches 9.6 MB +3.6 KB 207
other 2.5 MB +17.2 KB 493
Total 538.2 MB +21.1 KB 9,693

What a checkout or worktree writes to disk, from git ls-tree (benches/repo-size/tree.ts). The dashboard's Repo size page has the full picture.

Effect alone: effect-minimal bundle 8.9 KB gzip, hand-written effect-http call 48.6 KB gzip; effect import 68.0 ms and 12.8 MB heap; bare mocked HTTP roundtrip 10.4 µs p50.

Bundled (from src/): resend, neon, github, fly-io, aws, cloudflare, gcp, azure, stripe. Built and probed (from lib/): resend, neon, github, fly-io.

Job time: bundle 6.9 s, build: tsrs 2.4 s, build: bundle-lib 38.2 s, build: strip-lib-docs 0.1 s, build 40.7 s, size 0.9 s, runtime 12.0 s, providers 4.9 s.

¹ Timings run on a different runner than main's, so 🟡 marks a large move that may still be noise. Sizes, heap and object counts are stable. Offline: every call hits a canned in-memory response. Full suite: node benches/sdks/run.ts. Job log

github-actions Bot added a commit that referenced this pull request Nov 15, 2026
@alchemy-version-bot

alchemy-version-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Install the packages built from this commit:

@distilled.cloud

pnpm install https://pkg.distilled.cloud/@distilled.cloud/acme/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/adyen/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/anthropic/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/apache-superset/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/archil/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/argocd/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/auth0/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/aws/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/aws4signer/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/axiom/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/azure/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/boat-dev/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/celld/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/chronosphere/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/clerk/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/clickhouse-cloud/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/cloudflare/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/coinbase/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/coinflow/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/convex/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/coolify/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/core/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/customerio/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/datadog/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/daytona/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/digitalocean/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/discord/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/docker/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/dokploy/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/doppler/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/e2b/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/elasticsearch/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/entire/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/exa/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/expo-eas/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/fly-io/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/forgejo/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/gcp/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/github/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/google-workspace/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/grafana/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/growthbook/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/gusto/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/hetzner/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/hostinger/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/huggingface/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/iceberg/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/infisical/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/inngest/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/intercom/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/kubernetes/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/launchdarkly/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/linear/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/mailchimp/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/meilisearch/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/mercury/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/metabase/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/modal/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/modrinth/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/mongodb-atlas/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/namespacelabs/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/neon/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/netbird/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/notion/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/npm/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/okta/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/onepassword/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/opencode/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/opensearch/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/ovh/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/paypal/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/plaid/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/planetscale/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/polar/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/polymarket/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/porkbun/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/posthog/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/prisma/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/proxmox/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/questrade/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/railway/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/redis-cloud/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/remote/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/render/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/resend/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/revenuecat/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/rivet/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/s2/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/s3/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/scaleway/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/sentry/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/slack/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/spacetimedb/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/squarespace/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/stackit/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/starburst-galaxy/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/steel/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/stripe/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/supabase/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/surrealdb/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/tailscale/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/technitium/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/temporal/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/tencentcloud/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/trigger-dev/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/turbopuffer/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/turso/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/typesense/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/unkey/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/vanta/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/vercel/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/whop/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/workos/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/xata/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/zendesk/pr:1244:9d73b41
pnpm install https://pkg.distilled.cloud/@distilled.cloud/zerossl/pr:1244:9d73b41

Published Nov 15, 2026, 7:04 PM UTC. Expires Nov 22, 2026, 7:04 PM UTC, extended while this pull request is open.

github-actions Bot added a commit that referenced this pull request Nov 15, 2026
github-actions Bot added a commit that referenced this pull request Nov 15, 2026
@Mkassabov
Mkassabov added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit ed0e92f Nov 15, 2026
19 checks passed
@Mkassabov
Mkassabov deleted the ci/compat-nightly branch November 15, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: test the effect peer range, Bun and workerd

1 participant