Skip to content

🧪 fix: Wait Out the Langfuse Load-Time Verification in Its Tests and Bring the Live Model Specs Journey Up to Date - #204

Merged
danny-avila merged 1 commit into
mainfrom
pr/fu-test-reliability
Oct 28, 2026
Merged

danny-avila merged 1 commit into
mainfrom
pr/fu-test-reliability

Conversation

@danny-avila

Copy link
Copy Markdown
Contributor

Pull Request Template

⚠️ Before Submitting a PR, Please Review:

  • Please ensure that you have thoroughly read and understood the Contributing Docs before submitting your Pull Request.

⚠️ Documentation Updates Notice:

  • Kindly note that documentation updates are managed in this repository: librechat.ai

Summary

This PR fixes two unreliable tests on main. In both cases the test was wrong and the panel was right, so the PR changes only test files: LangfuseRenderer.test.tsx, ModelSpecsRenderer.test.tsx and e2e/specs.spec.ts.

LangfuseRenderer.test.tsx raced the verification the renderer starts on load.

"preserves the stored secret when only the public key is edited" failed once in CI with Unable to find an accessible element with the role "button" and name "com_config_langfuse_save_and_enable", then passed on re-run. Here is the async step it raced:

  • When LibreChat reports a saved connection, a useEffect in LangfuseRenderer verifies it once the status has rendered (testMutation.mutate). While that request is in flight (testMutation.isPending), every control is disabled and the primary button reads "Checking" instead of "Save & enable".
  • The test clicked Edit as soon as the Edit button appeared. At that moment the status had rendered, but the verification effect had not run yet, or had not settled. A click flushes pending effects, so the render after Edit showed "Checking", and the synchronous getByRole for Save & enable failed. When the effect had already committed its in-flight render, the Edit button was disabled, and the click did nothing.
  • The mock resolved at once, inside microtasks, so the window was usually too short to hit. It was hit only under load. Running the file in 8 concurrent loops on main failed 2 of 48 runs with the CI error. The failure output shows the public key edit open and the Save button reading com_config_langfuse_checking, disabled.

The other tests in the file:

  • "enables a disabled connection when credential edits are saved with Save & enable" had the same race: it waited only for the secret key preview, and it failed in the same stress runs.
  • "disables / enables a saved connection without re-verifying credentials" waited for mockTest to have been called once. That is already true while the request is in flight, so these two passed only because the in-flight render usually had not committed when they clicked.
  • "preserves draft fields when shared connection data refreshes" and "re-verifies the stored connection when an invalid key edit is cancelled" already waited for Verified.

The fix:

  • renderVerified(status) renders a saved connection and waits for the Verified status. Verified renders in the same pass in which the request is no longer pending, so the controls are enabled by then. It also asserts that exactly one verification ran. Every test that acts on a saved connection uses it (six tests), and their assertions are unchanged.
  • The default verification mock now answers on a later task (setTimeout(0)), as a real request does. The in-flight window is therefore always there, and a test that acts inside it fails on every run, not on one run in twenty under load. With this mock alone and the old test bodies, exactly the two tests that flaked fail, 5 runs out of 5.
  • A new test, "holds the connection controls until the stored connection is verified", pins the async step itself. With the verification held open, the status reads Checking and Edit public key, Edit secret key and Disable are disabled. Once the verification resolves, all three are enabled.
  • The neighbours: Langfuse is the only renderer under configuration/sections/ that runs queries or mutations, so no other section test races a request. The ones that do wait for something (mcp/Servers.test.tsx, mcp/Create.test.tsx and files/FileConfigRenderer.test.tsx) already await their dialogs and comboboxes with findByRole, and a stress run of all of them found no flakes (see Testing).

e2e/specs.spec.ts: the live Model specs journey expected an editor order the panel dropped.

"a new spec asks for Endpoint and Model up front and saves at the end of the list" failed live even when it ran alone on a clean database. toBeFocused on the new card's Endpoint select got "inactive", and the serial describe then skipped the remaining 8 journeys. The new card had focused its Label input. This is test drift, not a panel bug:

  • The journey was first written expecting a new spec to open on Label.
  • During the Wave 2 integration (✅ fix: Wave 2 Integration, Verification Fixes and Schema Coverage Guarantees #170), "a new spec opens on Endpoint and Model" made a new spec show its Model group first, and the journey followed ("test(e2e): a new spec opens on Endpoint").
  • A review then found that new and saved specs showed their groups in different orders, and that renaming a saved spec swapped the order mid-typing. The next day, "a new spec opens with the same group order as a saved one (Basics, then Model)" restored one order for both, and renamed the unit test "opens a new spec with Endpoint and Model up front, marked required" to "Add spec appends an empty spec". Nobody updated the journey.
  • A new entry card focuses the first field of its panel (ObjectEntryCard). With Basics first, that field is Label. The current Editor.tsx comment says the same: "Basics and Model (with the required name, label, endpoint and model) open first". Both groups are open on a new spec, with Endpoint and Model marked required.

With the focus expectation fixed, the same test failed on a second stale assertion that the first failure had been hiding. Since "spec rows name the endpoint", which landed hours after the journey was written, the card summary names the endpoint by its display name: e2e-spec · OpenAI · gpt-4o-mini. The test still expected the raw key, openAI · gpt-4o-mini.

The fix:

  • addSpec expects Label to be focused, then fills Label, Endpoint and Model in the order the card shows them. The summary assertion expects OpenAI · gpt-4o-mini.
  • The live journeys don't run in CI, which is how the drift went unnoticed. So ModelSpecsRenderer.test.tsx gains "opens a new spec on its Label, with Basics before Model as on a saved spec", which runs in CI. With the editor's groups swapped back to Model first, it fails. The stateful harness from the reorder test moved to module level, so both tests share it. The rename test and the new test share a helper that reads the group order.

Not in this PR

  • If the journey fails after its Save, the saved spec stays in Base, because revertList only runs once every assertion has passed. A later run then derives e2e-spec-2 and fails. That happened while I was debugging this PR, never in the final runs.
  • "a profile shows the specs it inherits" still skips on a database without a USER role profile. That is its documented precondition.

Change Type

  • Bug fix (non-breaking change which fixes an issue)

Testing

Unit tests, LangfuseRenderer.test.tsx

Run Result
main, 20 sequential runs 20 passed
main, 48 runs in 8 concurrent loops 2 failed: "preserves the stored secret…" and "enables a disabled connection…", each with the CI error
New on-next-task mock with the old test bodies, 5 runs Both tests failed in all 5 runs; every other test passed
This branch, 30 sequential runs 30 passed (16 tests each)
This branch, 80 runs in 10 concurrent loops 80 passed
This branch and a copy of main's file in the same 60 runs, in 12 concurrent loops plus 16 busy-loop CPU burners This branch: 60 passed. main's copy: 4 failed (the same two tests, twice each)
All 37 test files under configuration/sections/ (715 tests) on this branch's final commit, 12 runs in 4 concurrent loops 12 passed

Unit tests, ModelSpecsRenderer.test.tsx: 32 passed. With Editor.tsx changed to render the Model group before Basics (not committed), the new test fails on the group order.

Live journeys, e2e/specs.spec.ts

All runs used E2E_LIVE=1 against a local LibreChat backend. A clean database here means the admin configs collection was emptied and the backend restarted: the backend caches the merged config, so emptying the collection alone still served the previous run's saved spec.

Code Data Result
main clean Failed at toBeFocused, got "inactive"; 8 journeys did not run. The focused element was the new card's Label input.
Focus fix only clean Failed at toContainText('openAI · gpt-4o-mini'); the card read e2e-spec · OpenAI · gpt-4o-mini
This branch, twice back to back clean 10 passed, 1 skipped both times. The skip was "a profile shows the specs it inherits", for lack of a USER profile.
This branch an empty USER role profile added 11 passed

After the runs, Base and USER held no overrides. Against the mock backend, the spec runs its setup and skips its 10 live journeys, as before.

Test Configuration:

  • npx tsc --noEmit, npx eslint src/ --max-warnings 0, Prettier on the changed files and bun run test:lint-rules (19 passed) are clean.
  • npx vitest run --maxWorkers=3: 162 files, 3514 tests passed.
  • bun run build succeeds.

Checklist

  • My code adheres to this project's style guidelines
  • I have performed a self-review of my own code
  • I have commented in any complex areas of my code
  • My changes do not introduce new warnings
  • I have written tests demonstrating that my changes are effective or that my feature works
  • Local unit tests pass with my changes

…Bring the Live Model Specs Journey Up to Date

LangfuseRenderer verifies a saved connection once its status renders, and
while that request is in flight every control is disabled and the primary
button reads "Checking" instead of "Save & enable". "preserves the stored
secret when only the public key is edited" and "enables a disabled connection
when credential edits are saved with Save & enable" clicked Edit as soon as
the stored connection showed, before that verification had run or settled,
so under load the click landed inside the in-flight window and the Save &
enable lookup failed. renderVerified() now renders a saved connection and
waits for the Verified status, which renders in the same pass that releases
the controls; every test that acts on a saved connection uses it, with its
assertions unchanged. The default verification mock answers on a later task,
as a request does, so the window is always there and a test that acts inside
it fails on every run instead of rarely. A new test holds the verification
open and checks that Edit and Disable stay disabled until it settles.

e2e/specs.spec.ts expected a new spec to open on Endpoint. That held only
while a new spec showed its Model group first; the editor then went back to
one group order for new and saved specs (Basics, then Model), so a new card
focuses its Label, and the journey was never updated. The same journey also
expected the card summary to read "openAI · gpt-4o-mini", but spec rows name
the endpoint ("OpenAI") since the change that made them do so. addSpec now
expects Label focused and fills Label, Endpoint and Model in that order, and
the summary assertion expects "OpenAI · gpt-4o-mini". A unit test pins the
new spec's group order and its Label focus, so CI catches the next change.
@danny-avila

Copy link
Copy Markdown
Contributor Author

@codex review

Pass 1. Head: fa57d4d

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T10:10:47.306741Z fa57d4d Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: fa57d4d14c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila
danny-avila merged commit 0ba49f5 into main Oct 28, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant