Skip to content

Commit 17aa1a5

Browse files
Run one list of end-to-end scenarios through the Vitest, Mocha and Jest plugins, so a selection bug in any of them fails CI the same way (#3)
* Turn the repository into a pnpm workspace with the Vitest plugin under packages/vitest The code both plugins use (fs and child recording, the git anchor, the Actions cache client, the shard hash) moves to packages/shared, which is never published: each plugin compiles it into its own dist/. Releases are cut per package, from <package>-v<version> tags. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Add @useblacksmith/mocha: run only the Mocha spec files a change affects A preload records import edges, fs reads, child processes, worker threads and socket endpoints per spec file and per test; a wrapper command and a .mocharc.cjs helper select from that record. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mocha: keep test fixtures off the CI job's real Actions cache, and print only JSON on stdout for affected --json On a Blacksmith runner ACTIONS_RESULTS_URL is set, so the fixture repositories (identical commits, identical keys) restored each other's records. With --json the cache log lines now go to stderr. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vitest: restore the git-ignored fixture file the first commit dropped test/fixture-affected/src/generated/value.json is tracked in useblacksmith/vitest despite the fixture's own .gitignore; the copy into this repository skipped it, which failed 68 tests in CI. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mocha: a lockfile or config change drops every stored test record The change reran everything, but a test the run left out (--grep, --bail) kept its old pass and was skipped next time. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vitest: bring in useblacksmith/vitest main up to v0.6.5 (shadow marked deprecated and dropped from the README) The first commit here was taken before vitest main had #26 and the v0.6.5 release. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mocha: remove the --shadow flag The Vitest plugin has deprecated its shadow option; the Mocha plugin should not start with one. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mocha: refuse an option the command does not know run --shadow, now removed, was silently ignored and ran the affected selection; so was any mistyped option. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Add @useblacksmith/jest A config helper that sets Jest's runtime, testSequencer and a reporter: the runtime records what each test file loads and reads, the sequencer drops the test files no change affects and assigns shards by path hash, the reporter merges the run into the record and saves it to the Actions cache. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * jest: record what the config and globalSetup load and read, paths that appear, and failures a -t run cannot clear Also honours --passWithNoTests on the command line, and treats Babel, SWC and TypeScript config files in any directory as dependencies of every test file, since transformers memoize them per process. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * jest: restore the repository's formatting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * jest: keep the recorded transformer configs when a run covers only some test files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * jest: preload the child recorder by file URL and refuse a project that names its own runtime Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * jest: resync the builtin ESM exports of node:fs after wrapping it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Run one list of end-to-end scenarios through all three plugins, from source and as packed, against a fake and the real Actions cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * e2e: fail the pack step when no tarball was written Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent c82945c commit 17aa1a5

12 files changed

Lines changed: 738 additions & 4 deletions

File tree

‎.github/workflows/e2e.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: E2E
2+
3+
# Each plugin as npm would install it, run on the shared scenarios against
4+
# this job's real Actions cache. The CI workflow runs the same scenarios from
5+
# source against a fake cache.
6+
on:
7+
push:
8+
branches: [main]
9+
pull_request:
10+
workflow_dispatch:
11+
12+
permissions:
13+
contents: read
14+
15+
concurrency:
16+
group: e2e-${{ github.ref }}
17+
cancel-in-progress: true
18+
19+
jobs:
20+
scenarios:
21+
strategy:
22+
fail-fast: false
23+
matrix:
24+
plugin: [vitest, mocha, jest]
25+
runs-on: blacksmith-2vcpu-ubuntu-2404
26+
steps:
27+
- uses: actions/checkout@v6
28+
with:
29+
persist-credentials: false
30+
- uses: actions/setup-node@v6
31+
with:
32+
node-version-file: .nvmrc
33+
- run: corepack enable # pnpm, at the version package.json pins
34+
- run: pnpm install --frozen-lockfile
35+
- name: Pack the plugin
36+
id: pack
37+
working-directory: packages/${{ matrix.plugin }}
38+
run: |
39+
pnpm pack --pack-destination "$RUNNER_TEMP/tarball"
40+
# Fails the step when nothing was packed: an empty path would run the source instead.
41+
tarball=$(ls "$RUNNER_TEMP"/tarball/*.tgz)
42+
echo "tarball=$tarball" >> "$GITHUB_OUTPUT"
43+
- name: Run the scenarios against the real Actions cache
44+
working-directory: packages/${{ matrix.plugin }}
45+
env:
46+
BLACKSMITH_E2E_CACHE: real
47+
BLACKSMITH_E2E_TARBALL: ${{ steps.pack.outputs.tarball }}
48+
run: pnpm run test:e2e

‎README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,4 +25,27 @@ pnpm run build
2525
pnpm test
2626
```
2727

28+
### End-to-end scenarios
29+
30+
`packages/shared/test/scenarios.ts` lists the cases every plugin must pass: a
31+
small sample project, a real run of the test runner, a change (a source file,
32+
`package.json`, the lockfile, the workflow file, a new or failing test), and
33+
which test files the next run executes. Each package's
34+
`test/scenarios.test.ts` runs the list with its own runner, once with the
35+
record on disk and once with the record travelling only through a cache. To
36+
cover a new case in all three plugins, add it to that list.
37+
38+
| Run | Plugin | Cache |
39+
| --- | --- | --- |
40+
| `pnpm test` (CI workflow) | this checkout's source | a fake Actions cache in the test process |
41+
| E2E workflow | the packed tarball, installed with npm | the job's real Actions cache |
42+
43+
To run the packed plugin locally against the fake cache:
44+
45+
```sh
46+
cd packages/jest
47+
pnpm pack --pack-destination /tmp/tarball
48+
BLACKSMITH_E2E_TARBALL=/tmp/tarball/useblacksmith-jest-0.1.0.tgz pnpm run test:e2e
49+
```
50+
2851
Releases are cut per package: `pnpm run release <vitest|mocha|jest> <major|minor|patch>`.

‎packages/jest/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@
2626
"build": "tsc -p tsconfig.build.json",
2727
"prepack": "npm run build",
2828
"check:types": "tsc --noEmit",
29-
"test": "node --test --test-concurrency=1 test/*.test.ts"
29+
"test": "node --test --test-concurrency=1 test/*.test.ts",
30+
"test:e2e": "node --test --test-concurrency=1 test/scenarios.test.ts"
3031
},
3132
"peerDependencies": {
3233
"jest": ">=29.0.0"
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
import path from 'node:path'
2+
import { test } from 'node:test'
3+
4+
import type { Driver } from '../../shared/test/project.ts'
5+
import { cases } from '../../shared/test/scenarios.ts'
6+
import { packageRoot } from './harness.ts'
7+
8+
const driver: Driver = {
9+
name: 'jest',
10+
packageRoot,
11+
style: 'cjs',
12+
prelude: '',
13+
runner: 'jest',
14+
source: path.join(packageRoot, 'src', 'index.ts'),
15+
cacheDir: 'node_modules/.cache/blacksmith-jest',
16+
config: (plugin) => ({
17+
'jest.config.js': `const { withBlacksmith } = require(${JSON.stringify(plugin)})\nmodule.exports = withBlacksmith({ watchman: false })\n`,
18+
}),
19+
command: (project, report) => [
20+
path.join(project.cwd, 'node_modules', 'jest', 'bin', 'jest.js'),
21+
'--json',
22+
`--outputFile=${report}`,
23+
],
24+
files: (report) =>
25+
(report as { testResults: { name: string }[] }).testResults.map(
26+
(result) => result.name,
27+
),
28+
}
29+
30+
for (const { name, run } of cases(driver)) test(name, run)

‎packages/mocha/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@
2626
"build": "tsc -p tsconfig.build.json",
2727
"prepack": "npm run build",
2828
"check:types": "tsc --noEmit",
29-
"test": "node --test --test-concurrency=1 test/*.test.ts"
29+
"test": "node --test --test-concurrency=1 test/*.test.ts",
30+
"test:e2e": "node --test --test-concurrency=1 test/scenarios.test.ts"
3031
},
3132
"peerDependencies": {
3233
"mocha": ">=10.0.0"
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
import { readFileSync } from 'node:fs'
2+
import path from 'node:path'
3+
import { test } from 'node:test'
4+
5+
import type { Driver } from '../../shared/test/project.ts'
6+
import { cases } from '../../shared/test/scenarios.ts'
7+
import { packageRoot } from './harness.ts'
8+
9+
const driver: Driver = {
10+
name: 'mocha',
11+
packageRoot,
12+
style: 'cjs',
13+
prelude: '',
14+
runner: 'mocha',
15+
source: path.join(packageRoot, 'src', 'cli.ts'),
16+
cacheDir: 'node_modules/.cache/blacksmith-mocha',
17+
config: () => ({}),
18+
command(project, report) {
19+
const plugin = path.join(project.cwd, 'node_modules/@useblacksmith/mocha')
20+
const cli = project.installed
21+
? path.join(
22+
plugin,
23+
(
24+
JSON.parse(
25+
readFileSync(path.join(plugin, 'package.json'), 'utf8'),
26+
) as { bin: Record<string, string> }
27+
).bin['blacksmith-mocha'] ?? '',
28+
)
29+
: this.source
30+
return [
31+
cli,
32+
'run',
33+
'--',
34+
'test/*.test.js',
35+
'--reporter',
36+
'json',
37+
'--reporter-option',
38+
`output=${report}`,
39+
]
40+
},
41+
files: (report) =>
42+
(report as { tests: { file: string }[] }).tests.map((entry) => entry.file),
43+
}
44+
45+
for (const { name, run } of cases(driver)) test(name, run)

‎packages/shared/test/fake-cache.ts‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
import { createServer, type IncomingMessage } from 'node:http'
2+
3+
/** In-memory stand-in for the Actions cache service's twirp API plus its blob URLs. */
4+
export async function fakeCacheService(): Promise<{
5+
url: string
6+
close(): void
7+
}> {
8+
const entries = new Map<string, { blob?: Buffer; final: boolean }>()
9+
let port = 0
10+
const read = (req: IncomingMessage) =>
11+
new Promise<Buffer>((resolve) => {
12+
const chunks: Buffer[] = []
13+
req.on('data', (chunk: Buffer) => chunks.push(chunk))
14+
req.on('end', () => resolve(Buffer.concat(chunks)))
15+
})
16+
const server = createServer(async (req, res) => {
17+
const url = new URL(req.url ?? '/', 'http://x')
18+
const body = await read(req)
19+
const json = (status: number, payload: unknown) => {
20+
res.writeHead(status, { 'content-type': 'application/json' })
21+
res.end(JSON.stringify(payload))
22+
}
23+
const blobUrl = (key: string) =>
24+
`http://127.0.0.1:${port}/blob/${encodeURIComponent(key)}`
25+
if (url.pathname.startsWith('/blob/')) {
26+
const key = decodeURIComponent(url.pathname.slice('/blob/'.length))
27+
if (req.method === 'PUT') {
28+
entries.set(key, { blob: body, final: false })
29+
res.writeHead(201).end()
30+
} else {
31+
const blob = entries.get(key)?.blob
32+
if (!blob) return void res.writeHead(404).end()
33+
res.writeHead(200).end(blob)
34+
}
35+
return
36+
}
37+
const method = url.pathname.split('/').pop()
38+
const request = JSON.parse(body.toString('utf8')) as {
39+
key: string
40+
restore_keys?: string[]
41+
size_bytes?: number
42+
}
43+
if (method === 'GetCacheEntryDownloadURL') {
44+
const final = [...entries].filter(([, e]) => e.final).map(([k]) => k)
45+
const match =
46+
final.find((k) => k === request.key) ??
47+
final
48+
.reverse()
49+
.find((k) => request.restore_keys?.some((p) => k.startsWith(p)))
50+
return match
51+
? json(200, {
52+
ok: true,
53+
signed_download_url: blobUrl(match),
54+
matched_key: match,
55+
})
56+
: json(200, { ok: false })
57+
}
58+
if (method === 'CreateCacheEntry') {
59+
return entries.get(request.key)?.final
60+
? json(200, { ok: false })
61+
: json(200, { ok: true, signed_upload_url: blobUrl(request.key) })
62+
}
63+
if (method === 'FinalizeCacheEntryUpload') {
64+
const entry = entries.get(request.key)
65+
if (!entry?.blob) return json(200, { ok: false })
66+
if (entry.blob.byteLength !== Number(request.size_bytes))
67+
return json(422, { msg: 'size mismatch' })
68+
entry.final = true
69+
return json(200, { ok: true })
70+
}
71+
json(404, { msg: `unknown ${method}` })
72+
})
73+
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve))
74+
port = (server.address() as { port: number }).port
75+
return { url: `http://127.0.0.1:${port}/`, close: () => void server.close() }
76+
}

0 commit comments

Comments
 (0)