Skip to content

test(ai-sdk-provider): pin array content normalization for Gemini - #1157

Merged
andrelandgraf merged 2 commits into
mainfrom
aileen/neon-dx/task/gauge-6c1bc1f98584
Nov 17, 2026
Merged

andrelandgraf merged 2 commits into
mainfrom
aileen/neon-dx/task/gauge-6c1bc1f98584

Conversation

@andrelandgraf

@andrelandgraf andrelandgraf commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Found by a coding-agent eval of AI Gateway onboarding (Claude Code with Claude Opus 5.5): the chat completions content shape differs by model.

Problem

@neon/ai-sdk-provider wraps the unified chat route's fetch with a normalizer that flattens an array message.content into a string. Without it, the AI SDK rejects the response. The module and its wiring in provider.ts were marked TODO(#308): a temporary gpt-oss workaround, to be deleted once gpt-oss returns a compliant shape. That condition has now been met: gpt-oss models return a string on /v1/chat/completions.

But the normalizer still does real work. On a live gateway branch, gemini-3-7-flash, gemini-3-8-flash, and gemini-4-argon return message.content as an array of { type: 'text', text, thoughtSignature } on every non-streaming call. Following the TODO would break generateText for those models for every AI SDK user. The provider is also the route the AI Gateway docs now recommend for getting plain text from those models with no extra code.

No unit test covered that shape. The only end-to-end check exercised gpt-oss, so it passed with or without the wrapper.

Solution

  • neon-harmony-normalize.ts and provider.ts: replace the TODO with what is true now. The module normalizes any array content on the chat route, and it stays until the gateway returns a string for every model. The comments now describe the Gemini shape next to the gpt-oss one. No runtime code changed.
  • sdk-compatibility.test.ts: generateText with AI SDK 6 and 7 returns the text from the Gemini array shape through a mocked fetch. It lives in the provider-level test file rather than the normalizer's own, so deleting the module along with its tests still fails CI.
  • e2e/gateway-matrix.e2e.test.ts: the normalization block now calls gemini-4-argon instead of gpt-oss-120b. Its stream case is renamed as a plain sanity check, since Gemini streams string deltas.

This belongs in the provider because the provider is the layer that absorbs the gateway's array shape for AI SDK consumers. The docs that point users to it are updated in the matching neondatabase/website change.

No changeset: only comments and tests change.

Verification

  • With the harmony wrapper removed from provider.ts, the two new tests fail with Invalid input: expected string, received array at choices[0].message.content. With the wrapper in place, they pass.
  • Live, on a throwaway branch in a smoke org (deleted afterwards): the renamed e2e block passed against gemini-4-argon, for both generateText and streamText. A separate run of generateText and streamText through the provider returned plain text for all seven models that return array content on the raw route: three Gemini and four Claude.
  • Checks: pnpm --filter '...[HEAD]...' --filter '!.' build, then npm_config_workspace_concurrency=1 VITEST_MIN_FORKS=1 VITEST_MAX_FORKS=4 pnpm --filter '...[HEAD]' --filter '!.' test:ci --testTimeout=60000 (13 files passed, 1 skipped; 132 tests passed). pnpm lint:ci passes; its one info notice is in an untouched CLI file.

Left open

  • The root cause is in the gateway: /v1/chat/completions returns non-spec array content for these Gemini models. Once it returns a string, this module can be deleted, and the new tests should then be rewritten against a compliant body.

Opened by Aileen (factory neon-dx).

@andrelandgraf

Copy link
Copy Markdown
Member Author

@andrelandgraf andrelandgraf left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aileen review: approve (test, low risk, b1ba6e9)

Verdict

Approve, low risk. The diff changes comments and tests in @neon/ai-sdk-provider and no runtime code. It replaces the TODO(#308) that said to delete the array content normalizer with the condition that keeps it, adds a regression test for the Gemini array shape, and points the live e2e normalization block at gemini-4-argon.

Evidence

  • The new tests guard the wrapper. With wrapFetchWithHarmonyNormalization removed from provider.ts, both new sdk-compatibility.test.ts cases fail with Invalid input: expected string, received array at choices[0].message.content. With the wrapper in place, they pass. The tree was restored after the check.
  • Routing matches the claim: gemini-4-argon resolves to the mlflow route (createChatModel), which is the only route wrapped by the normalizer. Claude goes through the Anthropic route, so the four Claude models in the description's live check never reach this code.
  • Local build of affected packages passes. In packages/ai-sdk-provider, vitest run --testTimeout=60000 gives 13 files passed and 1 skipped (neon-catalog-drift, env-gated, pre-existing), with 132 tests passed. The root pnpm --filter '...[HEAD]' form matched no projects on a clean checkout, so the package suite was run directly.
  • PR checks are green on head b1ba6e9, including Live AI Gateway e2e, which runs the renamed gemini-4-argon block against a real gateway.

Regressions

There are none, because runtime behavior is unchanged. The only loss is the live streaming check for gpt-oss, which the description says now returns string content.

Non-blocking notes

  • The comments mention history ("gpt-oss, which it was written for (#308), is compliant now") and repeat the three Gemini ids in the module header and in the test fixture. Those lists will drift.
  • Nothing fails when the gateway is fixed: the e2e generateText case passes whether or not the wrapper rewrote anything, so deleting the module still depends on someone remembering to do it.
  • The streamText case in the normalization block is a plain smoke test, as its comment says.
  • maxOutputTokens: 512 on gemini-4-argon could flake if thinking uses up the budget. CI passed with it on this head.
  • The harmony naming is now a misnomer. Renaming it belongs in a separate PR.

@andrelandgraf

Copy link
Copy Markdown
Member Author

Waiting for Andre: category test needs Andre. Merge it when it is ready, or comment to have Aileen change it.

@andrelandgraf
andrelandgraf merged commit c221772 into main Nov 17, 2026
9 checks passed
@andrelandgraf
andrelandgraf deleted the aileen/neon-dx/task/gauge-6c1bc1f98584 branch November 17, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant