Skip to content
Draft
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
packages/cli/src/commands/_generated/** linguist-generated
packages/cli/src/__tests__/regression/** text eol=lf
packages/cli/src/__tests__/regression/snapshots/** linguist-generated
44 changes: 44 additions & 0 deletions .github/workflows/regression.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: HTTP regression

# Snapshots every HTTP request made by the cf command tree. The suite has 12
# balanced files; each GitHub Actions shard runs one file.

on:
pull_request:
merge_group:
types: [checks_requested]
branches:
- main
push:
branches:
- main

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
regression:
if: ${{ github.repository_owner == 'cloudflare' }}
name: regression (${{ matrix.shard }}/12)
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 1
persist-credentials: false

- name: Install dependencies
uses: ./.github/actions/install-dependencies

- name: Snapshot cf commands
run: pnpm --filter cf test:regression --shard=${{ matrix.shard }}/12
1 change: 1 addition & 0 deletions packages/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@
"build:test": "vp run task:build:test",
"check:type": "vp run task:check:type",
"test": "vp run task:test",
"test:regression": "vp test run --config regression.config.ts",
"test:watch": "vp run task:test:watch",
"test:imports": "vp run task:test:imports"
},
Expand Down
22 changes: 22 additions & 0 deletions packages/cli/regression.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import { defineConfig } from "vite-plus";
import base from "./vite.config.js";

export default defineConfig({
resolve: base.resolve,
test: {
...base.test,
env: { TZ: "UTC", CI: "1", CF_SEND_TELEMETRY: "0" },
include: ["src/__tests__/regression/shards/*.test.ts"],
exclude: [],
server: {
deps: {
inline: [
"@cloudflare/workers-utils",
"@cloudflare/workers-auth",
"@cloudflare/containers-shared",
"@cloudflare/deploy-helpers",
],
},
},
},
});
59 changes: 59 additions & 0 deletions packages/cli/src/__tests__/regression/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# HTTP request regression suite

From the repository root:

```sh
pnpm --filter cf test:regression
pnpm --filter cf test:regression -t 'cf zones list$'
pnpm --filter cf test:regression -t 'cf zones list$' -u
pnpm --filter cf test:regression --shard=1/12
```

Install against the frozen lockfile and run `pnpm generate` first when preparing
a fresh checkout. The suite has twelve command files balanced by case count.
Locally, Vitest runs them in parallel worker forks. CI runs the separate
`.github/workflows/regression.yml` workflow as twelve GitHub Actions shards,
each running one file. Each fork owns its temporary working directory, MSW
interceptor, credentials and state, and each command owns its snapshot.
Arguments after `test:regression` go directly to Vitest. The harness checks in
`harness.test.ts` run with the normal CLI test suite.

`cases.ts` discovers executable commands from the tracked command catalogue and
intermediate command groups from the generated source. It omits `cf auth`,
`cf build`, `cf deploy` and `cf dev`, including the hidden `cf login` alias, because their
credential and local-project setup adds complexity without useful request-shape
coverage. It also omits `cf containers build`, `cf containers push` and
`cf containers ssh`, which start Docker or SSH, and all `cf access` commands plus
`cf tunnels diag`, `login`, `quick-start`, `ready`, `run` and `tail`, which start
cloudflared. Every remaining leaf and group
runs without arguments. Leaves also run
with required arguments, individual optional flags, all enum choices, both
boolean values, pairwise flag presence/absence, alternate positional values,
aliases, global flags, local mode, raw/file/array bodies and derived body fields.
Equivalent argument vectors are deduplicated. Pairwise coverage is deliberately
bounded rather than an exponential Cartesian product; it covers flag presence,
not every combination of values. Invalid combinations remain in the baseline
with their exit code and error, alongside successful requests.

The real CLI parser, command modules, SDK and HTTP clients run against MSW.
Invocations use deterministic dummy API-token credentials in the environment.
`workflows.ts` supplies original local project fixtures and response shapes for
multi-request workflows. Generated commands must reach their declared method
and endpoint, and API-backed
hand-written workflows must reach their main endpoint. Project subprocesses, installs, native profiling and the Miniflare process are
replaced at their OS boundary. Local request rewriting remains real. These
snapshots cover the CLI's HTTP behavior, not those external programs.

Snapshots under `snapshots/<command/path>.snap` contain named cases, complete
arguments, exit outcomes and ordered requests. Requests preserve method, full
URL/query encoding, headers, JSON structure, binary bytes and ordered multipart
parts (including repeated names, file names and MIME types). Only CLI/Node
versions, temporary/fixture paths, random multipart boundaries and multipart
content lengths (which depend on those paths/boundaries) are normalized. Part
contents remain recorded. Fixtures use LF on every OS. No live account is needed.

On failure, Vitest names the command and snapshot file and shows the changed
request beside its case and arguments. Inspect the diff before using `-u`;
review method, URL, authentication/other headers and serialized body. If a new
command cannot reach HTTP, fix its fixture arguments or workflow responses
before accepting a baseline. Commit the reviewed snapshots with the change.
Loading
Loading