ci(e2e): add a curated @smoke gating leg, make the full generic leg informational#8759
Open
jacekradko wants to merge 3 commits into
Open
ci(e2e): add a curated @smoke gating leg, make the full generic leg informational#8759jacekradko wants to merge 3 commits into
jacekradko wants to merge 3 commits into
Conversation
🦋 Changeset detectedLatest commit: db5acc0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Repository UI (inherited) Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
The staging e2e "generic" leg was red on ~100% of runs because a few independent failures sat behind an all-or-nothing gate: - whatsapp-phone-code: the WhatsApp channel is not enabled on the staging instance, so the button never renders and the suite times out every run. It also bypasses the isStagingReady graceful-skip, so skip it explicitly on staging until the channel is provisioned. - custom-pages "survives a parent rerender": validates an unreleased @clerk/react fix (#8604), but the staging leg installs published @latest, so it is deterministically red until release. Skip when E2E_SDK_SOURCE=latest; PR CI (ref builds) still covers it. - concurrency was keyed on ref (effectively always "main") with cancel-in-progress, so each new staging deploy cancelled the in-flight run and no commit could report a status. Key on the clerk_go commit instead. - raise the job timeout above the 25-minute test step so the job cap no longer kills runs mid-suite. - emit and upload a JSON Playwright report in CI so the report job can classify failures (flaky vs failed, infra vs regression) later.
validate-staging-instances.mjs already diffs prod vs staging /v1/environment but every exit path returned 0, so detected drift blocked nothing and the job was not a dependency of the test matrix. A drifted staging mirror (e.g. a missing phone_number WhatsApp channel) therefore surfaced only as opaque test timeouts 200 tests deep. Add a tight CRITICAL_PATHS allowlist (attribute enabled toggles, phone_number.channels, auth factors/strategies, social enable/disable, password settings) and an ACCEPTED_DRIFT escape hatch so known gaps don't block while new drift does. In strict mode the script exits non-zero on a blocking mismatch; fetch failures and cosmetic drift never fail the build. Wire integration-tests to need validate-instances, and drive strictness from the STAGING_VALIDATE_STRICT repo variable (default report-only). So this is a no-op until the team opts in: it logs blocking drift and the proposed gate without failing anything. Flip the variable to make it enforce.
jacekradko
force-pushed
the
jacek/staging-e2e-validate-gate
branch
from
07c335c to
0eb5396
Compare
…nformational The generic leg runs ~210 tests against one shared staging instance, so any single post-retry flake reds the whole matrix and the staging run has been ~100% red, giving no signal. Split the gate: tag a handful of stable core-auth happy paths @smoke and run them as their own small matrix leg against just the two core apps, and mark the full generic leg continue-on-error so it still runs and uploads results but no longer fails the run. Because only gating legs (smoke + the framework legs) can fail the matrix job, needs.integration-tests.result is 'failure' only on a real gating-leg failure, so the report job's Slack notification fires on meaningful failures instead of on generic-only flake. The initial @smoke set is intentionally conservative (sign-in with password / instant password / modal, sign-up with password, sign-out) and is just tags, so it can grow as flake is brought under control.
jacekradko
force-pushed
the
jacek/staging-e2e-smoke-leg
branch
from
3d2c6ea to
db5acc0
Compare
jacekradko
force-pushed
the
jacek/staging-e2e-validate-gate
branch
from
bf10aa7 to
9353469
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
genericleg runs ~210 tests against one shared staging instance, so any single post-retry flake reds the whole matrix. That is the main reason the staging run has been ~100% red and gives no signal.This splits the gate. A handful of stable core-auth happy paths are tagged
@smokeand run as their own small matrix leg against just the two core apps, and the fullgenericleg becomescontinue-on-errorso it still runs and uploads results but no longer fails the run. Because only gating legs (smoke plus the framework legs) can fail the matrix job,needs.integration-tests.resultisfailureonly on a real gating-leg failure, so the existing report job notifies on meaningful failures instead of on generic-only flake. The clerk_go commit-status block stays commented out; when it is wired up (follow-up) it inherits the same gating semantics.The load-bearing choice here is the
@smokeset, since it is what gates. I kept it deliberately conservative: five happy paths that never appeared in a failed/flaky list across the recent runs (sign-in with password, with instant password, the modal variant, sign-up with password, and a single-flow sign-out), run againstreact.vite.withEmailCodesandnext.appRouter.withEmailCodes. It is just tags, so it should grow as the rate-limit flake is brought under control. A few tests now run in both legs (smoke and the informational generic), which is cheap insurance and lets the smoke result come from the clean, low-load leg rather than the noisy 210-test one.Stacked on #8757.