Conversation
Refactor all Cypress functional tests in a 1:1 migration to Playwright.
The CI runners have Yarn 1.x globally but the project uses Yarn 4.9.4 via packageManager field. Adding corepack enable ensures the correct Yarn version is used. Also removes unnecessary cd into subdirectory for functional_all_db.yml since yarn workspace commands run from root.
|
I did a lot of work on Playwright previously in Orchard Core Contrib (OCC). Moreover, I attempted to utilize Playwright from xUnit, which is much easier than TypeScript Thanks for this initiative, it could be a good start before integrating Playwright using .NET |
|
Nothing prevents from having both. |
|
Why not C# instead of TS? |
|
I could say because it's a 1:1 migration but that would not be a strong argument. As I said previously to Hisham, nothing prevents using both. For some reason, C# is always a strong point but for this migration it seemed easier to simply move it to its equivalent. |
|
In general, TypeScript is considered the superior choice for Playwright because the tool was built natively for the Node.js ecosystem. However, C# is a strong alternative if your team is already deeply integrated into the .NET platform. [1, 2, 3, 4]
Why Choose C#?
Key Comparison
Verdict: Use TypeScript for the most "complete" Playwright experience with the latest features. Choose C# if you want your tests to live in the same world as your .NET backend or existing C# test suites. [1, 16, 17, 18] [1] [https://club.ministryoftesting.com](https://club.ministryoftesting.com/t/is-playwright-c-worth-using/83635) |
|
Playwright has full support for C#, and the OC codebase is largely C#. These tests are not client-side code, so I don't see why we'd then go with TS, as opposed to all the other tests, which are also C# (and using xUnit). The AI-generated arguments don't really stand here. |
|
I'm using it for testing the new media gallery. Which makes sense since it is a VueJs application. I don't see any issue having both kind of tests. My goal is to not waste too much time on refactoring these tests in C# either. I just want to have tests for TUS protocol "resuming uploads" in the Media Gallery. I think, it is just a question of tastes which I won't debate on. Playwright allows for both, I do not have a favor between C# or Typescript. I just know that the tool is built with Typescript first so to me it makes sense having all kinds of tests with it. |
|
I think we should try to migrate this PR to C#, if not just to simplify how people run these tests, and hopefully write some (nobody has written a JS one in 10 years) |
|
@copilot Can you refactor this PR by using C# for the functional test instead. You can rename the project .csproj and folder to OrchardCore.Tests.Functional |
There was a problem hiding this comment.
Pull request overview
Migrates the repository’s end-to-end functional test suite from Cypress to Playwright, updating local tooling and CI workflows to run the new Playwright-based tests.
Changes:
- Added a new Playwright workspace (
test/OrchardCore.Tests.Playwright) with configs, helpers, fixtures, and initial CMS/MVC specs. - Removed the legacy Cypress functional test suite and its custom Cypress command package.
- Updated docs, solution/workspaces, gitignore, and GitHub Actions workflows to run Playwright tests (and moved Node to v20 in CI).
Reviewed changes
Copilot reviewed 67 out of 80 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| yarn.lock | Adds Playwright dependencies and workspace entry. |
| test/OrchardCore.Tests.Playwright/tsconfig.json | TypeScript config for Playwright tests/helpers. |
| test/OrchardCore.Tests.Playwright/tests/mvc/mvc.spec.ts | MVC smoke spec migrated to Playwright. |
| test/OrchardCore.Tests.Playwright/tests/cms/saas.spec.ts | SaaS recipe functional specs in Playwright. |
| test/OrchardCore.Tests.Playwright/tests/cms/saas-setup.spec.ts | Default tenant SaaS setup spec for CMS run ordering. |
| test/OrchardCore.Tests.Playwright/tests/cms/migrations.spec.ts | Migrations recipe specs in Playwright. |
| test/OrchardCore.Tests.Playwright/tests/cms/headless.spec.ts | Headless recipe specs in Playwright. |
| test/OrchardCore.Tests.Playwright/tests/cms/comingsoon.spec.ts | ComingSoon recipe specs in Playwright. |
| test/OrchardCore.Tests.Playwright/tests/cms/blog.spec.ts | Blog recipe specs in Playwright. |
| test/OrchardCore.Tests.Playwright/tests/cms/agency.spec.ts | Agency recipe specs in Playwright. |
| test/OrchardCore.Tests.Playwright/playwright.config.ts | Playwright runner configuration and project definitions. |
| test/OrchardCore.Tests.Playwright/package.json | New Yarn workspace package for Playwright tests. |
| test/OrchardCore.Tests.Playwright/helpers/utils.ts | Shared test utilities (tenant/config helpers). |
| test/OrchardCore.Tests.Playwright/helpers/urls.ts | URL navigation helper(s). |
| test/OrchardCore.Tests.Playwright/helpers/tenants.ts | Tenant creation/setup helpers for tests. |
| test/OrchardCore.Tests.Playwright/helpers/selectors.ts | data-cy selector helpers (Cypress parity). |
| test/OrchardCore.Tests.Playwright/helpers/recipes.ts | Recipe run/import helpers. |
| test/OrchardCore.Tests.Playwright/helpers/features.ts | Feature enable/disable helpers. |
| test/OrchardCore.Tests.Playwright/helpers/configuration.ts | Site configuration helper(s). |
| test/OrchardCore.Tests.Playwright/helpers/buttons.ts | Shared button-click helpers. |
| test/OrchardCore.Tests.Playwright/helpers/auth.ts | Login helper for Orchard admin. |
| test/OrchardCore.Tests.Playwright/helpers/app-lifecycle.ts | Build/host/ready-check lifecycle utilities for CI/local runs. |
| test/OrchardCore.Tests.Playwright/global-teardown.ts | Stops hosted app and cleans migrations recipe. |
| test/OrchardCore.Tests.Playwright/global-setup.ts | Hosts the app and prepares test environment. |
| test/OrchardCore.Tests.Playwright/fixtures/migrations.recipe.json | Adds migrations recipe fixture for tests. |
| test/OrchardCore.Tests.Playwright/OrchardCore.Tests.Playwright.csproj | Adds placeholder project to include Playwright test assets in solution. |
| test/OrchardCore.Tests.Functional/test_generator.js | Removes Cypress-era fixture generator. |
| test/OrchardCore.Tests.Functional/package.json | Removes Cypress functional test package config. |
| test/OrchardCore.Tests.Functional/mvc-tests/cypress/support/index.js | Removes Cypress MVC support bootstrap. |
| test/OrchardCore.Tests.Functional/mvc-tests/cypress/plugins/index.js | Removes Cypress MVC plugins entry. |
| test/OrchardCore.Tests.Functional/mvc-tests/cypress/integration/mvc-test.js | Removes Cypress MVC spec. |
| test/OrchardCore.Tests.Functional/mvc-tests/cypress.json | Removes Cypress MVC config. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/utils.js | Removes Cypress command utils source. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/urls.js | Removes Cypress URL command. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/test-runner.js | Removes Cypress test runner implementation. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/tenants.js | Removes Cypress tenant commands. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/recipe.js | Removes Cypress recipe commands. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/index.js | Removes Cypress commands index. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/features.js | Removes Cypress feature commands. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/cySelectors.js | Removes Cypress data-cy selector commands. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/configuration.js | Removes Cypress configuration commands. |
| test/OrchardCore.Tests.Functional/cypress-commands/src/buttons.js | Removes Cypress button commands. |
| test/OrchardCore.Tests.Functional/cypress-commands/rollup.config.js | Removes rollup bundling config for Cypress commands package. |
| test/OrchardCore.Tests.Functional/cypress-commands/package.json | Removes Cypress commands package manifest. |
| test/OrchardCore.Tests.Functional/cypress-commands/package-lock.json | Removes Cypress commands package lockfile. |
| test/OrchardCore.Tests.Functional/cypress-commands/dist/utils.mjs | Removes built Cypress utils artifact. |
| test/OrchardCore.Tests.Functional/cypress-commands/dist/utils.js | Removes built Cypress utils artifact. |
| test/OrchardCore.Tests.Functional/cypress-commands/dist/test-runner.mjs | Removes built Cypress test runner artifact. |
| test/OrchardCore.Tests.Functional/cypress-commands/dist/test-runner.js | Removes built Cypress test runner artifact. |
| test/OrchardCore.Tests.Functional/cypress-commands/dist/index.mjs | Removes built Cypress command bundle. |
| test/OrchardCore.Tests.Functional/cypress-commands/dist/index.js | Removes built Cypress command bundle. |
| test/OrchardCore.Tests.Functional/cypress-commands/README.md | Removes Cypress commands package docs. |
| test/OrchardCore.Tests.Functional/cypress-commands/LICENSE | Removes Cypress commands package license file. |
| test/OrchardCore.Tests.Functional/cypress-commands/.npmignore | Removes Cypress commands publish ignore file. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress/support/index.js | Removes Cypress CMS support bootstrap. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress/plugins/index.js | Removes Cypress CMS plugins entry. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress/integration/sass-test.js | Removes Cypress SaaS spec. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress/integration/migrations-test.js | Removes Cypress migrations spec. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress/integration/headless-test.js | Removes Cypress headless spec. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress/integration/comingsoon-test.js | Removes Cypress coming soon spec. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress/integration/blog-test.js | Removes Cypress blog spec. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress/integration/agency-test.js | Removes Cypress agency spec. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress/integration/000-saas-setup.js | Removes Cypress SaaS setup bootstrap spec. |
| test/OrchardCore.Tests.Functional/cms-tests/cypress.json | Removes Cypress CMS config. |
| test/OrchardCore.Tests.Functional/Readme.md | Removes Cypress functional test documentation. |
| test/OrchardCore.Tests.Functional/OrchardCore.Tests.Functional.csproj | Removes Cypress functional tests placeholder project. |
| test/OrchardCore.Tests.Functional/.gitignore | Removes Cypress-specific ignores (now handled at repo root). |
| src/docs/reference/libraries/README.md | Updates test library list from Cypress to Playwright. |
| package.json | Adds workspace + root scripts to run Playwright tests. |
| OrchardCore.slnx | Replaces Cypress functional test project with Playwright one. |
| AGENTS.md | Updates agent guidance to run Playwright functional tests. |
| .gitignore | Ignores Playwright reports and test-results. |
| .github/workflows/release_ci.yml | Runs Playwright functional tests + uploads reports on failure. |
| .github/workflows/preview_ci.yml | Runs Playwright functional tests + uploads reports on failure. |
| .github/workflows/pr_ci.yml | Runs Playwright functional tests + uploads reports on failure. |
| .github/workflows/main_ci.yml | Runs Playwright functional tests + uploads reports on failure. |
| .github/workflows/functional_all_db.yml | Updates functional test matrix workflow to Playwright and Node 20. |
Files not reviewed (1)
- test/OrchardCore.Tests.Functional/cypress-commands/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
* Initial plan * Refactor Playwright functional tests from TypeScript to C# - Remove TypeScript Playwright test project (test/OrchardCore.Tests.Playwright) - Create C# Playwright test project at test/OrchardCore.Tests.Functional - Add xUnit v3 test classes mirroring all original TS tests - Add helper classes for auth, tenants, configuration, lifecycle, etc. - Update all CI workflows to use dotnet test instead of yarn/npm - Update OrchardCore.slnx, Directory.Packages.props, AGENTS.md - Revert root package.json (remove Playwright workspace/scripts) Co-authored-by: Skrypt <3228637+Skrypt@users.noreply.github.com> * Fix code review feedback: use base-32 encoding and static members Co-authored-by: Skrypt <3228637+Skrypt@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Skrypt <3228637+Skrypt@users.noreply.github.com>
|
@copilot Fix the Github CI workflow that is failing. |
|
https://github.com/OrchardCMS/OrchardCore/actions/runs/23652309194/job/68900142655 |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 68 out of 80 changed files in this pull request and generated 5 comments.
Files not reviewed (1)
- test/OrchardCore.Tests.Functional/cypress-commands/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Replace AppLifecycleHelper recipe file copy/delete with EmbeddedRecipeHarvester that serves test recipes directly from embedded assembly resources via IRecipeHarvester — no filesystem writes, no shared lock, no ownership tracking. Any .recipe.json added to the Fixtures/ folder is discovered automatically. Remove all SetEnvironmentVariable calls from OrchardTestServer. Database configuration is now injected per-host via builder.Configuration (which takes priority over env vars), making each host self-contained with zero process-global side effects. Replace SqliteConnection.ClearAllPools() with a targeted ClearPool() scoped to the fixture's specific database file. These changes remove every shared-mutable-state blocker, enabling full parallel execution of functional test fixtures.
|
@copilot This last commit seems to make the parallel execution even worst because of tenant initialization time. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 68 out of 80 changed files in this pull request and generated 3 comments.
Files not reviewed (1)
- test/OrchardCore.Tests.Functional/cypress-commands/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ter.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Replace Substring concatenation with string.Concat and AsSpan to satisfy the CA1845 analyzer rule.
Authenticate PostgreSQL and MySQL service container image pulls with DOCKERHUB_USERNAME/DOCKERHUB_PASSWORD secrets to avoid Docker Hub anonymous rate limits (100 pulls/6h) on shared GitHub Actions runners.
ShellSettingsManager re-adds EnvironmentVariablesConfigurationProvider with highest priority, overriding the per-fixture connection strings set via builder.Configuration. All fixtures ended up using the same database, causing 'Table already exists' errors under parallel execution. Replace static readonly capture with CaptureAndClear that reads the CI env vars once at static initialization and immediately removes them. Each fixture's database config is then applied exclusively through tenants.json and builder.Configuration without env var interference.
Fix two issues that caused SaaS functional tests to fail on PostgreSQL, MySQL, and SQL Server: 1. CreateTenantAsync skipped setting the table prefix when DatabaseConfigurationPreset was true (external DB on parent), because the #DatabaseProvider dropdown was not rendered and the entire block was guarded by its presence. Move table prefix handling outside the dropdown check so it always runs when the field is visible. 2. EnsureDatabaseExists kept leftover tables from previous runs, causing "database, table prefix and schema are already in use" validation errors. Replace with EnsureCleanDatabase that drops and recreates the fixture database. Uses pg_terminate_backend for Postgres < 13 compatibility. Also add run-db-tests.sh helper script to run functional tests against all database providers locally via Docker, and force English locale so test assertions match on non-English machines. Update README with usage.
Describe CaptureAndClear pattern instead of claiming no env vars are mutated, note that fixture databases are dropped and recreated for clean state, and clarify that maxParallelThreads defaults to CPU core count when omitted from xunit.runner.json.
|
Full parallel build now working: |
Summary
cypress-commandspackage, npm dependencies (package.json,package-lock.json), JS test files, and the customtest_generator.jsOrchardTestFixture,OrchardTestServer,EmbeddedRecipeHarvester,TenantHelper,AuthHelper,FeatureHelper,ButtonHelper,SelectorHelper,ConfigurationHelper, andTestUtilsEmbeddedRecipeHarvester(implementsIRecipeHarvester), eliminating shared filesystem state — any.recipe.jsonadded toFixtures/is discovered automaticallyCaptureAndClearand injected per-host throughbuilder.Configurationandtenants.jsonDatabaseConfigurationPresethid the provider dropdown — now set independently of dropdown visibilitySharedViewCompilerProviderstatic-to-instance field change to prevent view cache corruption when multiple test hosts run in-processInMemoryLoggerProviderwithFakeLoggerProviderfromMicrosoft.Extensions.Diagnostics.Testingmain_ci,pr_ci,preview_ci,release_ci,functional_all_db): remove Node.js setup steps, add Playwright browser caching, align artifact upload paths with per-fixtureApp_Data_Tests_*directories, remove deadORCHARD_APPenv var, and add Docker Hub credentials for service container image pullsrun-db-tests.shhelper script to run functional tests locally against all database providers via DockerTest plan
run-db-tests.sh allfunctional_all_dbCI workflow to confirm parallel execution across database providersSharedViewCompilerProviderstatic→instance change)🤖 Generated with Claude Code