Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions content/docs/ai-gateway/chat-completions.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ summary: >-
Gateway. It is OpenAI Chat Completions-compatible, works with any model in
the catalog, and lets you switch providers without changing your SDK code.
enableTableOfContents: true
updatedOn: '2026-10-23T11:00:38.670Z'
updatedOn: '2026-11-17T23:43:15.556Z'
---

The chat completions endpoint is the recommended way to use Neon AI Gateway. It's fully compatible with the [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat) and works with every model in the [AI Gateway catalog](/docs/ai-gateway/models). Switch models by changing a single field.
Expand Down Expand Up @@ -158,7 +158,7 @@ model: 'gemini-3-flash'
model: 'qwen3-next-80b-a3b-instruct'
```

For a few models, `message.content` comes back as an array of content blocks instead of a plain string. See [Content shape varies by model](/docs/ai-gateway/models#which-endpoint-to-use) before swapping in a model you haven't used yet.
For a few Claude and Gemini models, `message.content` comes back as an array of content blocks instead of a plain string. [Content shape varies by model](/docs/ai-gateway/models#which-endpoint-to-use) lists those models and how to get plain text from them.

Gemini models don't support `parallel_tool_calls: false`, and a request that sets it returns `400` with `Unknown name "parallel_tool_calls"`. Leave the field out instead. Gemini makes parallel tool calls by default, so your code needs to handle more than one tool call per turn.

Expand Down
4 changes: 2 additions & 2 deletions content/docs/ai-gateway/get-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ summary: >-
host, and making your first request to the Neon AI Gateway using the OpenAI
SDK. No provider API keys required. Authenticate with your Neon credential.
enableTableOfContents: true
updatedOn: '2026-10-01T22:38:59.908Z'
updatedOn: '2026-11-17T18:07:26.865Z'
---

To set up Neon AI Gateway with an AI coding assistant, install the Neon Platform (`neon`) and Neon AI Gateway skills with the [Neon CLI](/docs/cli):
Expand Down Expand Up @@ -209,7 +209,7 @@ curl -X POST "$NEON_AI_GATEWAY_BASE_URL/v1/chat/completions" \

## Swap models

Change the `model` field to use a different provider. No other code changes required.
Change the `model` field to use a different provider. For most models, no other code changes are required. A few Claude and Gemini models return `message.content` as an array instead of a string. See [Content shape varies by model](/docs/ai-gateway/models#which-endpoint-to-use) for the list.

```typescript
// OpenAI
Expand Down
46 changes: 37 additions & 9 deletions content/docs/ai-gateway/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ summary: >-
models, which endpoint to use, shorter and longer request paths, the
/v1/models API, rate limits, pricing, and provider terms.
enableTableOfContents: true
updatedOn: '2026-11-15T17:20:24.920Z'
updatedOn: '2026-11-17T23:45:15.648Z'
---

This page covers how to call models through Neon AI Gateway and the rules that apply. For the full list of models you can call, see the [model catalog](https://neon.com/ai-gateway#models).
Expand Down Expand Up @@ -83,17 +83,45 @@ All paths below are appended to your branch's bare AI Gateway host (`NEON_AI_GAT
| TypeSafe AI (decision models) | `/typesafe/v1/systemone` | No chat completions. Returns answers to questions, not text. See [Decisions](/docs/ai-gateway/decisions) |

<Admonition type="warning" title="Content shape varies by model">
For most models, `message.content` in a chat completions response is a plain string. For `claude-opus-5-5`, it's an array of typed content blocks once the model reasons (`{ type: 'reasoning', ... }`, `{ type: 'text', text: ... }`); a trivial prompt still returns a string. For `gemini-4-argon`, it's an array of `text` blocks, even for a trivial prompt. A low `max_tokens` value can also cut a response off before the `text` block appears, leaving only a `reasoning` block. Handle both shapes:
On `/v1/chat/completions`, `message.content` is a plain string for most models. For the models below, the gateway returns an array of typed content blocks instead. This is a known gateway issue that we're fixing.

- `gemini-3-7-flash`, `gemini-3-8-flash`, `gemini-4-argon`, and `gemini-4-carbon` return an array on every non-streaming response, even for a trivial prompt. Their streamed `delta.content` values are strings, with `null` on the final chunk.
- `claude-opus-5-5`, `claude-haiku-5-5`, `claude-fable-5-1`, and `claude-fable-5-2` return a string for a trivial prompt and an array once the model reasons. When they reason while streaming, some `delta.content` values are arrays too. A low `max_tokens` value can cut a response off before the `text` block appears, leaving only a `reasoning` block.

The arrays have these shapes:

```typescript
// Gemini models
[
{ type: 'text', text: '...', thoughtSignature: '...' },
];

// Claude models, once they reason
[
{ type: 'reasoning', summary: [...] },
{ type: 'text', text: '...' },
];
```

To get plain text from every model without extra code, call the gateway through [`@neon/ai-sdk-provider`](https://www.npmjs.com/package/@neon/ai-sdk-provider) with the Vercel AI SDK. The native endpoints return a fixed shape: [Gemini](/docs/ai-gateway/gemini) through the google-genai SDK gives you `response.text`, and [Anthropic Messages](/docs/ai-gateway/anthropic-messages) always returns an array of content blocks. With the OpenAI SDK on `/v1/chat/completions`, accept both shapes and join the `text` blocks:

```typescript
// The OpenAI SDK types content as string | null, so read it as unknown.
const content: unknown = response.choices[0].message.content;
const text =
typeof content === 'string'
? content
: Array.isArray(content)
? (content.find((block) => block?.type === 'text')?.text ?? '')
: '';
function textOf(content: unknown): string {
if (typeof content === 'string') return content;
if (!Array.isArray(content)) return '';
return content
.filter((block) => block?.type === 'text' && typeof block.text === 'string')
.map((block) => block.text)
.join('');
}

const text = textOf(response.choices[0].message.content);

// Streamed deltas can be arrays too, so read them the same way.
for await (const chunk of stream) {
process.stdout.write(textOf(chunk.choices[0]?.delta?.content));
}
```

</Admonition>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "What is the best way to call models from multiple LLM providers with one API key?"
subtitle: "Use Neon AI Gateway: one Neon credential and one OpenAI-compatible endpoint reach models from OpenAI, Anthropic, Google, and open-weight labs, and you switch models by changing the model string."
createdAt: '2026-11-02T00:00:00.000Z'
updatedOn: '2026-11-03T23:21:47.530Z'
updatedOn: '2026-11-17T18:07:26.865Z'
isDraft: false
previousLink:
title: 'What is the best way to add OAuth to an MCP server?'
Expand Down Expand Up @@ -56,7 +56,7 @@ Chat completions is the recommended starting point, but the same credential also
- **Prepaid credits**: AI Gateway draws from [prepaid credits](/docs/ai-gateway/prepaid-credits) ($5 minimum purchase). Every model in the catalog is available once you have prepaid credits.
- **Regions**: your project must be in AWS US East (Ohio), US East (N. Virginia), Europe (Frankfurt), or Asia Pacific (Singapore), with support expanding toward [all regions](/docs/introduction/regions).
- **Rate limits**: soft limits of 200,000 input and 200,000 output tokens per minute by default, counted separately, though this may vary by account ([rate limits](/docs/ai-gateway/models#rate-limits)).
- **Response shape**: a few models return `message.content` as an array of content blocks instead of a string, so handle both when you swap in a new model.
- **Response shape**: a few Claude and Gemini models return `message.content` as an array of content blocks instead of a string. Check the [list](/docs/ai-gateway/models#which-endpoint-to-use) before you swap one in, or call models through `@neon/ai-sdk-provider`, which returns plain text for every model.
- **Data handling**: most models run on Databricks Model Serving, which may store inputs and outputs for up to 30 days to detect abuse ([data retention](/docs/ai-gateway/data-retention)).

## How other options compare
Expand Down
4 changes: 2 additions & 2 deletions content/faqs/test-different-llm-model-preview-branch.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "How do I test a different LLM model on a preview branch before switching production?"
subtitle: "Branch your Neon backend, deploy the branch with a different model ID through Neon AI Gateway, and run your evals against a copy-on-write clone of production data. Production keeps its model until you merge the change."
createdAt: '2026-11-02T00:00:00.000Z'
updatedOn: '2026-11-03T23:21:47.530Z'
updatedOn: '2026-11-17T18:07:26.865Z'
isDraft: false
previousLink:
title: 'Is there a Supabase Realtime alternative that supports live queries with joins and aggregates?'
Expand Down Expand Up @@ -64,7 +64,7 @@ To keep eval branches from piling up, give new branches a TTL with the `branch`

## Things to check when you swap models

- **Response shape**: for most models `message.content` is a string, but some return an array of content blocks, for example once a model reasons. Handle both shapes ([which endpoint to use](/docs/ai-gateway/models#which-endpoint-to-use)).
- **Response shape**: for most models `message.content` is a string, but a few Claude and Gemini models return an array of content blocks, some only once they reason. If the model you're switching to is on the [list](/docs/ai-gateway/models#which-endpoint-to-use), handle both shapes or call it through `@neon/ai-sdk-provider`, which returns plain text.
- **Unsupported parameters**: Gemini models reject `parallel_tool_calls: false`, and `gemini-4-argon` rejects `temperature` and `reasoning_effort` on chat completions ([chat completions](/docs/ai-gateway/chat-completions#switching-models)).
- **Endpoint requirements**: Codex variants such as `gpt-5-3-codex` work only on the [Responses API endpoint](/docs/ai-gateway/openai-responses).
- **Cost**: eval traffic draws down the same prepaid credit balance as production, at each provider's per-token rate. Compare per-model rates in [`neon.com/models.json`](https://neon.com/models.json) before you switch.
Expand Down
10 changes: 6 additions & 4 deletions src/app/models/capabilities.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"$comment": "Measured behaviour of every model the Neon AI Gateway serves. Produced by calling each model on /v1/chat/completions, its provider-native dialect, and /openai/v1/responses with the web_search and image_generation tools. This is what the gateway DID; models.json is what each model CLAIMS. /models needs both to know which code examples are correct. The `embeddings: true` entries (qwen3-embedding-0-6b, gte-large-en, bge-large-en) are a different shape — POST /v1/embeddings, not chat — and were verified by hand against a live gateway branch (the first two on 2026-09-25, bge-large-en on 2026-10-13), not by probe:conformance, which only exercises the chat-shaped routes above. `chat: \"not-served\"` / `nativeDialect: \"none\"` on those three is accurate (chat_completions and openai_responses both reject them by name), not a placeholder. The `decision: true` entries (openjev-qwen35-4b, typesafeai-jev-1-14-0) are System One decision models served only on POST /typesafe/v1/systemone; they were verified by hand against a production gateway branch on 2026-10-22 (chat_completions rejects them by name, so `chat: \"not-served\"` is accurate).",
"$howToRegenerate": "Generated from the conformance record in neondatabase/neon-console-code-examples: run `bun run probe:conformance` there, then copy its conformance.json into the shape below. One probe feeds both repos so the console and /models cannot disagree. Fields per model: id, owner, entitled, chat (conforms | array-content | not-served | not-entitled), deviations[], nativeDialect (anthropic | openai-responses | gemini | none), responses, webSearch, imageGeneration. Conformance MUST be measured with a prompt that makes the model reason as well as a trivial one, taking the worse shape: claude-opus-5, claude-sonnet-5 and claude-fable-5 return a compliant string for a trivial prompt and an array of {reasoning, text} parts once they think, and an easy-prompt-only probe is what published all three as `conforms`. CI checks completeness, not freshness - see the models-rest validator in scripts/verify-agent-endpoints.mjs.",
"$comment": "Measured behaviour of every model the Neon AI Gateway serves. Produced by calling each model on /v1/chat/completions, its provider-native dialect, and /openai/v1/responses with the web_search and image_generation tools. This is what the gateway DID; models.json is what each model CLAIMS. /models needs both to know which code examples are correct. The `embeddings: true` entries (qwen3-embedding-0-6b, gte-large-en, bge-large-en) are a different shape — POST /v1/embeddings, not chat — and were verified by hand against a live gateway branch (the first two on 2026-09-25, bge-large-en on 2026-10-13), not by probe:conformance, which only exercises the chat-shaped routes above. `chat: \"not-served\"` / `nativeDialect: \"none\"` on those three is accurate (chat_completions and openai_responses both reject them by name), not a placeholder. The `decision: true` entries (openjev-qwen35-4b, typesafeai-jev-1-14-0) are System One decision models served only on POST /typesafe/v1/systemone; they were verified by hand against a production gateway branch on 2026-10-22 (chat_completions rejects them by name, so `chat: \"not-served\"` is accurate). claude-fable-5-1 was re-measured by hand against a live gateway branch with a reasoning prompt: it returns an array of {reasoning, text} parts on /v1/chat/completions once it thinks, so it is `array-content` here ahead of the next probe:conformance run, and the console's conformance record still lists it as `conforms` until that run.",
"$howToRegenerate": "Generated from the conformance record in neondatabase/neon-console-code-examples: run `bun run probe:conformance` there, then copy its conformance.json into the shape below. One probe feeds both repos so the console and /models cannot disagree. Fields per model: id, owner, entitled, chat (conforms | array-content | not-served | not-entitled), deviations[], nativeDialect (anthropic | openai-responses | gemini | none), responses, webSearch, imageGeneration. Conformance MUST be measured with a prompt that makes the model reason as well as a trivial one, taking the worse shape: claude-opus-5-5, claude-haiku-5-5 and claude-fable-5-1 return a compliant string for a trivial prompt and an array of {reasoning, text} parts once they think, and an easy-prompt-only probe publishes such a model as `conforms`. When the record changes which models are `array-content`, update the list in the \"Content shape varies by model\" admonition in content/docs/ai-gateway/models.md; src/app/models/content-shape-docs.test.js fails until they match. CI checks completeness, not freshness - see the models-rest validator in scripts/verify-agent-endpoints.mjs.",
"probedAt": "2026-10-23T16:24:29.630Z",
"models": [
{
Expand All @@ -18,8 +18,10 @@
"id": "claude-fable-5-1",
"owner": "anthropic",
"entitled": true,
"chat": "conforms",
"deviations": [],
"chat": "array-content",
"deviations": [
"`content` is an array[2] of {reasoning, text} with keys {summary, text, type}; the spec requires a string"
],
"nativeDialect": "anthropic",
"responses": false,
"webSearch": false,
Expand Down
40 changes: 40 additions & 0 deletions src/app/models/content-shape-docs.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import fs from 'fs';
import path from 'path';

import { describe, it, expect } from 'vitest';

import capabilities from './capabilities.json';

// The models page tells readers which models need array handling on chat completions.
// It is prose, so nothing regenerates it when capabilities.json changes.
const modelsDoc = fs.readFileSync(
path.resolve(__dirname, '../../../content/docs/ai-gateway/models.md'),
'utf-8'
);

const admonition = () => {
const start = modelsDoc.indexOf('title="Content shape varies by model"');
const end = modelsDoc.indexOf('</Admonition>', start);
expect(start).toBeGreaterThan(-1);
return modelsDoc.slice(start, end);
};

describe('"Content shape varies by model" in the AI Gateway models doc', () => {
it('names exactly the models measured as array-content', () => {
const servedIds = new Set(capabilities.models.map((m) => m.id));
const named = [...admonition().matchAll(/`([a-z0-9-]+)`/g)]
.map(([, id]) => id)
.filter((id) => servedIds.has(id));
const measured = capabilities.models.filter((m) => m.chat === 'array-content').map((m) => m.id);

expect([...new Set(named)].sort()).toEqual(measured.sort());
});

it('points at the provider and the native endpoints', () => {
const text = admonition();

expect(text).toContain('@neon/ai-sdk-provider');
expect(text).toContain('/docs/ai-gateway/anthropic-messages');
expect(text).toContain('/docs/ai-gateway/gemini');
});
});
2 changes: 1 addition & 1 deletion src/app/models/examples.js
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ print(resp.choices[0].message.content)
});

// cURL swaps to a provider-native dialect only when the unified route is the broken one.
// A native dialect on its own is not a reason to leave `/v1/chat/completions` — the eight
// A native dialect on its own is not a reason to leave `/v1/chat/completions` — the
// conforming Claude models are served at /anthropic/v1/messages too, and showing that
// instead would trade a portable snippet for a vendor-specific one to fix nothing.
const curlChat = (model, nativeDialect, responsesOnly, arrayContent) => {
Expand Down
13 changes: 13 additions & 0 deletions src/app/models/route.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,19 @@ describe('GET /models', () => {
expect(curl.variantReason).toBeTruthy();
});

// It returns a string for a trivial prompt, so only a reasoning probe catches it.
it('serves no OpenAI SDK chat examples for claude-fable-5-1, which returns array content once it reasons', async () => {
const res = await GET(request('?model=claude-fable-5-1'));
const { model } = await body(res);
const ids = model.examples.map((e) => e.id);
const curl = model.examples.find((e) => e.id === 'curl');

expect(model.capabilities.chat).toBe('array-content');
expect(ids).not.toContain('typescript');
expect(ids).not.toContain('python');
expect(curl.endpoint).toBe('/anthropic/v1/messages');
});

it('keeps cURL on chat completions for a conforming Claude model', async () => {
const res = await GET(request('?model=claude-haiku-4-5'));
const { model } = await body(res);
Expand Down
Loading