adding playwright VRT integration by rigoucr · Pull Request #86 · fourkitchens/fire

rigoucr · 2026-05-27T16:56:59Z

Functional Testing Instructions
Run these from the Drupal project root.

Install the dev version from this PR branch:

composer remove fourkitchens/fire
composer require fourkitchens/fire:"dev-feature/playwright-vrt" --dev --prefer-source
./vendor/fourkitchens/fire/bin/fire --version

Set the Playwright test root:
export ATK_HOME=./tests/playwright
Run the Playwright init command:
./vendor/fourkitchens/fire/bin/fire vrt:init
At the prompts, enter test values such as:

BASELINE_URL="https://mysite.com"
CANDIDATE_URL="http://mylocalsite.lndo.site"
BASELINE_TERMINUS_ENV=live
BASELINE_TERMINUS_SITE=my-site

Verify generated files exist:

ls tests/playwright/.env
ls tests/playwright/package.json
ls tests/playwright/playwright.config.js
ls tests/playwright/playwright.atk.config.js

Verify .env content:

cat tests/playwright/.env
Expected:
BASELINE_URL="https://mysite.live"
CANDIDATE_URL="http://mysite.lndo.site"
BASELINE_TERMINUS_ENV=live
BASELINE_TERMINUS_SITE=my-site

Verify npm scripts were added:

cd tests/playwright
npm pkg get scripts

Expected scripts include:

{
  "vrt": "playwright-vrt",
  "vrt:local": "playwright-vrt-local",
  "vrt:ci": "playwright-vrt-ci"
}

Verify Playwright VRT runs through FIRE:

cd ../..
./vendor/fourkitchens/fire/bin/fire vrt:run --tool=playwright
11. Verify reference command uses the Playwright npm script:
./vendor/fourkitchens/fire/bin/fire vrt:reference --tool=playwright
12. Verify automatic tool detection:
./vendor/fourkitchens/fire/bin/fire vrt:run
./vendor/fourkitchens/fire/bin/fire vrt:reference

Expected:

If only Playwright is configured, FIRE runs Playwright automatically.
If both Backstop and Playwright are configured, FIRE asks which tool to run.
With --no-interaction, FIRE defaults to Playwright when both are configured.

Optional direct npm verification:

cd tests/playwright
npm run vrt

laura-johnson · 2026-06-17T15:22:11Z

+ */
+require('dotenv').config();
+
+const configuredBaseURL = (process.env.REMOTE_ENV_BASE_URL || '__DEFAULT_BASE_URL__')


It looks like this file reads process.env.REMOTE_ENV_BASE_URL to set the base URL. But the .env file the command generates writes CANDIDATE_URL and BASELINE_URL Mismatch?

laura-johnson · 2026-06-17T15:51:10Z

+```
+
+The `.env` file is added to `tests/playwright/.gitignore` by the init command.
+


I think we need to add a part here where we describe how to edit the common pages.

nJim · 2026-07-02T00:37:01Z

@@ -0,0 +1,46 @@
+const { devices } = require('@playwright/test')


We're mixing 'require' and 'export' in the same file. We should pick one standard.

nJim · 2026-07-02T00:38:28Z

+
+    const images = Array.from(document.querySelectorAll('img[loading="lazy"]'))
+
+    for (const image of images) {


FWIW, for loops can be slow sequence. Since this is JS we can fire these off in parallel and just await the work to be done.

nJim · 2026-07-02T02:12:36Z

+      $this->taskExec($command)->run();
+    }
+    else {
+      $this->taskExec($env . ' npm install')->dir($testsRoot)->run();


I'm struggling to trace these commands back, including the init/install and reference commands. Some of them run in DDEV and other run on NPM. This worked for me because I already had playwright installed locally, but I'm not sure the logic is sound here. Maybe someone else can test from a clean install.

nJim · 2026-07-02T02:22:34Z

+    $drupalRoot = $this->getDrupalRoot();
+    $atkHome = getenv('ATK_HOME');
+    if (!$atkHome) {
+      throw new AbortTasksException('ATK_HOME is not set in your environment. Add `export ATK_HOME=./tests/playwright` to your shell profile (PATH) and rerun.');


I'm a little confused by this. We are telling them to set a path, but it has to be this specific path. Can we simply hardcode the path? If not, should we remove the hardcoded value in the 'run' command?

nJim

Approving the work, but added a number of comments to consider as future changes. In general I would say the direction is good. Backstop is aging and Playwright is a better long term fit. I also understand wanting to put this into Fire to help with adoption.

I had trouble getting this running locally, which surfaced how many moving parts there are and how hard failures are to debug. We have:

composer required modules
ATK to run scaffolding
FIRE to copy templates, set ENVs, update files
NPM to install packages and browsers
CLI scripts that sometimes call other CLI script

I think this orchestration is going to make this very difficult to roll out. A proven template (tests/playwright) could simply be copied to every project with a readme to explain how to use it.

I think this effort looks like it's more about the automation of setup than the testing itself. The configuration does not exactly look correct (timeout, retry, possibly recording all traces.) Some variables are inconsistent (hardcoded vs configurable, ATK_HOME only works for init but not run). I think we can get this a little tighter before pushing for a every-site rollout. Or maybe try one site at a time and see what we learn.

Great work on moving this project forward. I think we can merge this as a big step towards the larger rollout.

nJim · 2026-07-02T14:18:08Z

+   * @option $y Run the command with no interection required.
   *
   */
  public function vrtInit(ConsoleIO $io) {


vrt:init calls vrt:playwright:init feels strange on the command line. If we are depricating backstop, why even support new installations?

Updating Fire commands - adding playwright VRT integration

38a911b

rigoucr changed the title ~~Updating Fire commands - adding playwright VRT integration~~ adding playwright VRT integration May 27, 2026

rigoucr added 3 commits May 28, 2026 15:42

Debuging and reshaping the the vrt init command

d2dac42

multiple improvements

0c4c209

Final details for Playwright + fire integration

4388bc3

rigoucr marked this pull request as ready for review June 16, 2026 15:58

rigoucr self-assigned this Jun 16, 2026

laura-johnson reviewed Jun 17, 2026

View reviewed changes

nJim reviewed Jul 2, 2026

View reviewed changes

nJim approved these changes Jul 2, 2026

View reviewed changes

nJim reviewed Jul 2, 2026

View reviewed changes

nJim added the passed review label Jul 2, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

adding playwright VRT integration#86

adding playwright VRT integration#86
rigoucr wants to merge 4 commits into
3.xfrom
feature/playwright-vrt

rigoucr commented May 27, 2026 •

edited

Loading

Uh oh!

laura-johnson Jun 17, 2026

Uh oh!

laura-johnson Jun 17, 2026

Uh oh!

nJim Jul 2, 2026

Uh oh!

nJim Jul 2, 2026

Uh oh!

nJim Jul 2, 2026

Uh oh!

nJim Jul 2, 2026

Uh oh!

nJim left a comment

Uh oh!

nJim Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		```

		The `.env` file is added to `tests/playwright/.gitignore` by the init command.

		@@ -0,0 +1,46 @@
		const { devices } = require('@playwright/test')


		const images = Array.from(document.querySelectorAll('img[loading="lazy"]'))

		for (const image of images) {

Uh oh!

Conversation

rigoucr commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laura-johnson Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

laura-johnson Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

nJim Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

nJim Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

nJim Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

nJim Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

nJim left a comment

Choose a reason for hiding this comment

Uh oh!

nJim Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rigoucr commented May 27, 2026 •

edited

Loading