Skip to content

docs: list the AI Gateway models that return array chat content - #6789

Merged
andrelandgraf merged 4 commits into
mainfrom
aileen/neon-dx/task/gauge-6c1bc1f98584
Nov 17, 2026
Merged

andrelandgraf merged 4 commits into
mainfrom
aileen/neon-dx/task/gauge-6c1bc1f98584

Conversation

@andrelandgraf

@andrelandgraf andrelandgraf commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Problem

On /v1/chat/completions, Neon AI Gateway returns message.content as an array of content blocks for some Claude and Gemini models instead of the string the OpenAI Chat Completions spec requires. It's a known gateway issue that's being fixed.

The "Content shape varies by model" warning in the AI Gateway models doc named only claude-opus-5-5 and gemini-4-argon, then told every reader to handle both shapes. Several models that return arrays weren't listed. The chat completions page, the get-started "Swap models" section and two FAQ pages repeated the blanket advice, and get-started said swapping models needs "no other code changes".

/models and the per-model docs pages pick their code examples from src/app/models/capabilities.json. That record listed claude-fable-5-1 as conforms, so its page served plain OpenAI SDK snippets that read message.content as a string. Those snippets break once the model reasons.

What the gateway returns

Measured live against /v1/chat/completions:

Models Non-streaming message.content Streaming delta.content
gemini-3-7-flash, gemini-3-8-flash, gemini-4-argon, gemini-4-carbon Array on every call Strings, null on the final chunk
claude-opus-5-5, claude-haiku-5-5, claude-fable-5-1, claude-fable-5-2 String for a trivial prompt, array once the model reasons Array deltas appear while the model reasons
claude-sonnet-5-5, gpt-5-mini String Strings

The array shapes:

// Gemini models
[{ type: 'text', text: '...', thoughtSignature: '...' }];

// Claude models, once they reason
[{ type: 'reasoning', summary: [...] }, { type: 'text', text: '...' }];

The docs change

The warning in content/docs/ai-gateway/models.md now lists the eight models above, says which return arrays on every call and which only once they reason, shows both shapes and states that this is a known gateway issue being fixed. It points readers to @neon/ai-sdk-provider for plain text from every model, and to the native Gemini and Anthropic Messages endpoints, which return a fixed shape.

For OpenAI SDK users, one helper replaces the old pair of snippets. It joins the text blocks and reads messages and streamed deltas the same way:

// The OpenAI SDK types content as string | null, so read it as unknown.
function textOf(content: unknown): string {
  if (typeof content === 'string') return content;
  if (!Array.isArray(content)) return '';
  return content
    .filter((block) => block?.type === 'text' && typeof block.text === 'string')
    .map((block) => block.text)
    .join('');
}

const text = textOf(response.choices[0].message.content);

// Streamed deltas can be arrays too, so read them the same way.
for await (const chunk of stream) {
  process.stdout.write(textOf(chunk.choices[0]?.delta?.content));
}

The old snippet took only the first text block and the old streaming snippet dropped every array delta. The helper keeps all text in both cases.

chat-completions.md, get-started.md and the two FAQ pages now say "a few Claude and Gemini models" and link to the list. Get-started says no other code changes are needed "for most models".

Before After
Before: warning names two models After: warning lists eight models, both shapes and one helper

Also in here

  • src/app/models/capabilities.json: claude-fable-5-1 is array-content, so /models and its docs page serve the Anthropic Messages cURL example instead of OpenAI SDK snippets. The $comment and $howToRegenerate notes name the Claude models that return arrays once they reason, and say to update the docs list when the record changes.
  • src/app/models/content-shape-docs.test.js (new): fails when the warning's model list disagrees with the array-content entries in capabilities.json, or when it stops linking the provider and both native endpoint pages.
  • src/app/models/route.test.js: claude-fable-5-1 gets no OpenAI SDK chat examples, and its cURL example uses /anthropic/v1/messages.
  • src/app/models/examples.js: a comment dropped a hard-coded model count that no longer held.

Verification

  • The textOf helper type-checks under tsc --strict and returns the right text for a string, null, the Gemini array, the Claude reasoning-plus-text array and a reasoning-only array.
  • npm run test:unit:run passes, including the new tests. npm run check:docs and npm run check:docs-api-consistency pass.
  • Prettier and markdownlint pass on every changed content file. The repo-wide check:md and lint:md runs report failures only in files this PR doesn't touch.
  • The models page renders correctly in light mode, dark mode and at a 390px width on a local dev server.
  • Not verified locally: the /docs/ai-gateway/models.md mirror, which is generated at build time.

For your attention

  • gemini-4-carbon and claude-fable-5-2 aren't in capabilities.json, so /models doesn't serve them and their docs pages show no code examples. Adding them needs measured values for every field (native dialect, Responses, web search, image generation), and that's a separate change. Until then, the new test can't check those two entries in the docs list.
  • The console reads the same conformance record from neondatabase/neon-console-code-examples, which still lists claude-fable-5-1 as conforms. The next bun run probe:conformance run there needs a reasoning prompt to pick it up.
  • Once the gateway returns a string for these models, the warning and the array-content entries can be removed.

@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
neon-next Ready Ready Preview Nov 17, 2026 11:48pm UTC

Request Review

@andrelandgraf

Copy link
Copy Markdown
Member Author

@andrelandgraf andrelandgraf left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aileen review: approve (fix, low risk, 49c0cd0)

Verdict

Approve at low risk. The change corrects the conformance record for two Claude models and the docs that describe it. Nothing touches auth, billing, migrations, or secrets.

What I verified

  • Live probe: I created a throwaway project in the Testing smoke org, probed /v1/chat/completions, then deleted the project.
    • On a reasoning prompt, claude-sonnet-5-5, claude-fable-5-1, claude-haiku-5-5, and claude-opus-5-5 returned array[reasoning,text].
    • claude-sonnet-5 returned a string. That matches the $howToRegenerate correction.
    • gemini-3-8-flash returned array[text], and gpt-5-mini returned a string.
    • The easier bat-and-ball prompt got strings from every Claude model. This confirms that the record has to be measured with a reasoning prompt.
  • Streaming: claude-sonnet-5-5 sent 2 array reasoning chunks and 120 string text chunks. gemini-4-argon sent string chunks and one final null. The new string-only streaming snippet keeps the full answer for both.
  • Provider claim: the claim that @neon/ai-sdk-provider returns plain text holds in neon-pkgs code. Claude routes to native Messages, and Gemini chat goes through wrapFetchWithHarmonyNormalization.
  • Effect on /models: claude-sonnet-5-5 and claude-fable-5-1 now get the same examples as claude-opus-5-5: AI SDK, Mastra, and Anthropic Messages cURL. They lose the OpenAI SDK TypeScript and Python snippets. Those snippets break once the model reasons, so this is intended. The .md mirrors are gitignored and generated at build time, so nothing needed regenerating.
  • Factory checks:
    • prettier --check passes on the 5 changed content files.
    • eslint reports one import/order warning in route.test.js. It is also present on main.
    • npm run test:unit:run -- --retry=2 --maxWorkers=4 --testTimeout=30000 --hookTimeout=60000 passed: 106 files, 1463 tests.
    • verify-agent-endpoints passed 68/68 checks, and check:models-endpoint-sync reports 63 models in sync.
    • check:docs-api-consistency found no broken references.

Non-blocking nits

From me and a second-opinion Claude Opus 5.5 review:

  • chat-completions.md:161 says "Every other model returns a string". models.md says "Every other chat model", which is more precise.

  • The Gemini streaming sentence in models.md leaves out the final-chunk null. The snippet already handles it.

  • content-shape-docs.test.js:

    • It ignores backticked ids that are missing from capabilities.json.
    • It would fail if a conforming model were named inside the admonition.
    • Its second test only checks link wording.

    This is acceptable for a guard on hand-maintained prose.

  • Already listed in the PR's "Left open" section:

    • The console's conformance record in neondatabase/neon-console-code-examples still lists both models as conforms.
    • The reason the Mastra example gives for using neon(...) is out of date.
    • The root cause is in the gateway.

@andrelandgraf

Copy link
Copy Markdown
Member Author

Waiting for Andre: category fix needs Andre. Merge it when it is ready, or comment to have Aileen change it.

…vior

Drop claude-sonnet-5-5 (returns a string), add gemini-4-carbon and
claude-fable-5-2, show both array shapes, and replace the OpenAI SDK
snippets with one helper that joins text blocks for messages and
streamed deltas.
@andrelandgraf andrelandgraf changed the title fix: name every AI Gateway model that returns array chat content docs: list the AI Gateway models that return array chat content Nov 17, 2026
@andrelandgraf
andrelandgraf merged commit e520dd1 into main Nov 17, 2026
5 checks passed
@andrelandgraf
andrelandgraf deleted the aileen/neon-dx/task/gauge-6c1bc1f98584 branch November 17, 2026 23:49

This branch was successfully deployed

1 active deployment
Preview – neon-next — 1537bd41 Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants