Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions content/docs/ai-gateway/chat-completions.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ summary: >-
Gateway. It is OpenAI Chat Completions-compatible, works with any model in
the catalog, and lets you switch providers without changing your SDK code.
enableTableOfContents: true
updatedOn: '2026-10-23T11:00:38.670Z'
updatedOn: '2026-11-17T18:07:26.865Z'
---

The chat completions endpoint is the recommended way to use Neon AI Gateway. It's fully compatible with the [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat) and works with every model in the [AI Gateway catalog](/docs/ai-gateway/models). Switch models by changing a single field.
Expand Down Expand Up @@ -158,7 +158,7 @@ model: 'gemini-3-flash'
model: 'qwen3-next-80b-a3b-instruct'
```

For a few models, `message.content` comes back as an array of content blocks instead of a plain string. See [Content shape varies by model](/docs/ai-gateway/models#which-endpoint-to-use) before swapping in a model you haven't used yet.
For a few Claude and Gemini models, `message.content` comes back as an array of content blocks instead of a plain string. Every other model returns a string. [Content shape varies by model](/docs/ai-gateway/models#which-endpoint-to-use) lists those models and how to get plain text from them.

Gemini models don't support `parallel_tool_calls: false`, and a request that sets it returns `400` with `Unknown name "parallel_tool_calls"`. Leave the field out instead. Gemini makes parallel tool calls by default, so your code needs to handle more than one tool call per turn.

Expand Down
4 changes: 2 additions & 2 deletions content/docs/ai-gateway/get-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ summary: >-
host, and making your first request to the Neon AI Gateway using the OpenAI
SDK. No provider API keys required. Authenticate with your Neon credential.
enableTableOfContents: true
updatedOn: '2026-10-01T22:38:59.908Z'
updatedOn: '2026-11-17T18:07:26.865Z'
---

To set up Neon AI Gateway with an AI coding assistant, install the Neon Platform (`neon`) and Neon AI Gateway skills with the [Neon CLI](/docs/cli):
Expand Down Expand Up @@ -209,7 +209,7 @@ curl -X POST "$NEON_AI_GATEWAY_BASE_URL/v1/chat/completions" \

## Swap models

Change the `model` field to use a different provider. No other code changes required.
Change the `model` field to use a different provider. For most models, no other code changes are required. A few Claude and Gemini models return `message.content` as an array instead of a string. See [Content shape varies by model](/docs/ai-gateway/models#which-endpoint-to-use) for the list.

```typescript
// OpenAI
Expand Down
20 changes: 18 additions & 2 deletions content/docs/ai-gateway/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ summary: >-
models, which endpoint to use, shorter and longer request paths, the
/v1/models API, rate limits, pricing, and provider terms.
enableTableOfContents: true
updatedOn: '2026-11-15T17:20:24.920Z'
updatedOn: '2026-11-17T18:07:26.865Z'
---

This page covers how to call models through Neon AI Gateway and the rules that apply. For the full list of models you can call, see the [model catalog](https://neon.com/ai-gateway#models).
Expand Down Expand Up @@ -83,7 +83,14 @@ All paths below are appended to your branch's bare AI Gateway host (`NEON_AI_GAT
| TypeSafe AI (decision models) | `/typesafe/v1/systemone` | No chat completions. Returns answers to questions, not text. See [Decisions](/docs/ai-gateway/decisions) |

<Admonition type="warning" title="Content shape varies by model">
For most models, `message.content` in a chat completions response is a plain string. For `claude-opus-5-5`, it's an array of typed content blocks once the model reasons (`{ type: 'reasoning', ... }`, `{ type: 'text', text: ... }`); a trivial prompt still returns a string. For `gemini-4-argon`, it's an array of `text` blocks, even for a trivial prompt. A low `max_tokens` value can also cut a response off before the `text` block appears, leaving only a `reasoning` block. Handle both shapes:
On `/v1/chat/completions`, `message.content` is a plain string for most models. These models return an array of typed content blocks instead:

- `gemini-3-7-flash`, `gemini-3-8-flash`, and `gemini-4-argon` return an array of `text` blocks on every non-streaming response, even for a trivial prompt. Their streamed `delta.content` values are strings.
- `claude-opus-5-5`, `claude-sonnet-5-5`, `claude-haiku-5-5`, and `claude-fable-5-1` return a string for a trivial prompt and an array of blocks (`{ type: 'reasoning', ... }`, `{ type: 'text', text: ... }`) once the model reasons. When streaming, their reasoning chunks carry an array in `delta.content` and their text chunks carry a string. A low `max_tokens` value can cut a response off before the `text` block appears, leaving only a `reasoning` block.

Every other chat model returns a string, so code that reads `message.content` as a string works with it unchanged.

To get plain text from the models above without extra code, call them through [`@neon/ai-sdk-provider`](https://www.npmjs.com/package/@neon/ai-sdk-provider) with the Vercel AI SDK, which returns text for every model. Their native endpoints return a fixed shape instead: [Gemini](/docs/ai-gateway/gemini) through the google-genai SDK gives you `response.text`, and [Anthropic Messages](/docs/ai-gateway/anthropic-messages) always returns an array of content blocks, so read the `text` blocks. With the OpenAI SDK on `/v1/chat/completions`, handle both shapes:

```typescript
// The OpenAI SDK types content as string | null, so read it as unknown.
Expand All @@ -96,6 +103,15 @@ const text =
: '';
```

When streaming, write only string deltas:

```typescript
for await (const chunk of stream) {
const delta: unknown = chunk.choices[0]?.delta?.content;
if (typeof delta === 'string') process.stdout.write(delta);
}
```

</Admonition>

## Shorter paths
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "What is the best way to call models from multiple LLM providers with one API key?"
subtitle: "Use Neon AI Gateway: one Neon credential and one OpenAI-compatible endpoint reach models from OpenAI, Anthropic, Google, and open-weight labs, and you switch models by changing the model string."
createdAt: '2026-11-02T00:00:00.000Z'
updatedOn: '2026-11-03T23:21:47.530Z'
updatedOn: '2026-11-17T18:07:26.865Z'
isDraft: false
previousLink:
title: 'What is the best way to add OAuth to an MCP server?'
Expand Down Expand Up @@ -56,7 +56,7 @@ Chat completions is the recommended starting point, but the same credential also
- **Prepaid credits**: AI Gateway draws from [prepaid credits](/docs/ai-gateway/prepaid-credits) ($5 minimum purchase). Every model in the catalog is available once you have prepaid credits.
- **Regions**: your project must be in AWS US East (Ohio), US East (N. Virginia), Europe (Frankfurt), or Asia Pacific (Singapore), with support expanding toward [all regions](/docs/introduction/regions).
- **Rate limits**: soft limits of 200,000 input and 200,000 output tokens per minute by default, counted separately, though this may vary by account ([rate limits](/docs/ai-gateway/models#rate-limits)).
- **Response shape**: a few models return `message.content` as an array of content blocks instead of a string, so handle both when you swap in a new model.
- **Response shape**: a few Claude and Gemini models return `message.content` as an array of content blocks instead of a string. Check the [list](/docs/ai-gateway/models#which-endpoint-to-use) before you swap one in, or call models through `@neon/ai-sdk-provider`, which returns plain text for every model.
- **Data handling**: most models run on Databricks Model Serving, which may store inputs and outputs for up to 30 days to detect abuse ([data retention](/docs/ai-gateway/data-retention)).

## How other options compare
Expand Down
4 changes: 2 additions & 2 deletions content/faqs/test-different-llm-model-preview-branch.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "How do I test a different LLM model on a preview branch before switching production?"
subtitle: "Branch your Neon backend, deploy the branch with a different model ID through Neon AI Gateway, and run your evals against a copy-on-write clone of production data. Production keeps its model until you merge the change."
createdAt: '2026-11-02T00:00:00.000Z'
updatedOn: '2026-11-03T23:21:47.530Z'
updatedOn: '2026-11-17T18:07:26.865Z'
isDraft: false
previousLink:
title: 'Is there a Supabase Realtime alternative that supports live queries with joins and aggregates?'
Expand Down Expand Up @@ -64,7 +64,7 @@ To keep eval branches from piling up, give new branches a TTL with the `branch`

## Things to check when you swap models

- **Response shape**: for most models `message.content` is a string, but some return an array of content blocks, for example once a model reasons. Handle both shapes ([which endpoint to use](/docs/ai-gateway/models#which-endpoint-to-use)).
- **Response shape**: for most models `message.content` is a string, but a few Claude and Gemini models return an array of content blocks, some only once they reason. If the model you're switching to is on the [list](/docs/ai-gateway/models#which-endpoint-to-use), handle both shapes or call it through `@neon/ai-sdk-provider`, which returns plain text.
- **Unsupported parameters**: Gemini models reject `parallel_tool_calls: false`, and `gemini-4-argon` rejects `temperature` and `reasoning_effort` on chat completions ([chat completions](/docs/ai-gateway/chat-completions#switching-models)).
- **Endpoint requirements**: Codex variants such as `gpt-5-3-codex` work only on the [Responses API endpoint](/docs/ai-gateway/openai-responses).
- **Cost**: eval traffic draws down the same prepaid credit balance as production, at each provider's per-token rate. Compare per-model rates in [`neon.com/models.json`](https://neon.com/models.json) before you switch.
Expand Down
16 changes: 10 additions & 6 deletions src/app/models/capabilities.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"$comment": "Measured behaviour of every model the Neon AI Gateway serves. Produced by calling each model on /v1/chat/completions, its provider-native dialect, and /openai/v1/responses with the web_search and image_generation tools. This is what the gateway DID; models.json is what each model CLAIMS. /models needs both to know which code examples are correct. The `embeddings: true` entries (qwen3-embedding-0-6b, gte-large-en, bge-large-en) are a different shape — POST /v1/embeddings, not chat — and were verified by hand against a live gateway branch (the first two on 2026-09-25, bge-large-en on 2026-10-13), not by probe:conformance, which only exercises the chat-shaped routes above. `chat: \"not-served\"` / `nativeDialect: \"none\"` on those three is accurate (chat_completions and openai_responses both reject them by name), not a placeholder. The `decision: true` entries (openjev-qwen35-4b, typesafeai-jev-1-14-0) are System One decision models served only on POST /typesafe/v1/systemone; they were verified by hand against a production gateway branch on 2026-10-22 (chat_completions rejects them by name, so `chat: \"not-served\"` is accurate).",
"$howToRegenerate": "Generated from the conformance record in neondatabase/neon-console-code-examples: run `bun run probe:conformance` there, then copy its conformance.json into the shape below. One probe feeds both repos so the console and /models cannot disagree. Fields per model: id, owner, entitled, chat (conforms | array-content | not-served | not-entitled), deviations[], nativeDialect (anthropic | openai-responses | gemini | none), responses, webSearch, imageGeneration. Conformance MUST be measured with a prompt that makes the model reason as well as a trivial one, taking the worse shape: claude-opus-5, claude-sonnet-5 and claude-fable-5 return a compliant string for a trivial prompt and an array of {reasoning, text} parts once they think, and an easy-prompt-only probe is what published all three as `conforms`. CI checks completeness, not freshness - see the models-rest validator in scripts/verify-agent-endpoints.mjs.",
"$comment": "Measured behaviour of every model the Neon AI Gateway serves. Produced by calling each model on /v1/chat/completions, its provider-native dialect, and /openai/v1/responses with the web_search and image_generation tools. This is what the gateway DID; models.json is what each model CLAIMS. /models needs both to know which code examples are correct. The `embeddings: true` entries (qwen3-embedding-0-6b, gte-large-en, bge-large-en) are a different shape — POST /v1/embeddings, not chat — and were verified by hand against a live gateway branch (the first two on 2026-09-25, bge-large-en on 2026-10-13), not by probe:conformance, which only exercises the chat-shaped routes above. `chat: \"not-served\"` / `nativeDialect: \"none\"` on those three is accurate (chat_completions and openai_responses both reject them by name), not a placeholder. The `decision: true` entries (openjev-qwen35-4b, typesafeai-jev-1-14-0) are System One decision models served only on POST /typesafe/v1/systemone; they were verified by hand against a production gateway branch on 2026-10-22 (chat_completions rejects them by name, so `chat: \"not-served\"` is accurate). claude-sonnet-5-5 and claude-fable-5-1 were re-measured by hand against a live gateway branch on 2026-11-17 with a reasoning prompt: both return an array of {reasoning, text} parts on /v1/chat/completions once they think, so they are `array-content` here ahead of the next probe:conformance run, and the console's conformance record still lists them as `conforms` until that run.",
"$howToRegenerate": "Generated from the conformance record in neondatabase/neon-console-code-examples: run `bun run probe:conformance` there, then copy its conformance.json into the shape below. One probe feeds both repos so the console and /models cannot disagree. Fields per model: id, owner, entitled, chat (conforms | array-content | not-served | not-entitled), deviations[], nativeDialect (anthropic | openai-responses | gemini | none), responses, webSearch, imageGeneration. Conformance MUST be measured with a prompt that makes the model reason as well as a trivial one, taking the worse shape: claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5 and claude-fable-5-1 return a compliant string for a trivial prompt and an array of {reasoning, text} parts once they think, and an easy-prompt-only probe publishes such a model as `conforms`. When the record changes which models are `array-content`, update the list in the \"Content shape varies by model\" admonition in content/docs/ai-gateway/models.md; src/app/models/content-shape-docs.test.js fails until they match. CI checks completeness, not freshness - see the models-rest validator in scripts/verify-agent-endpoints.mjs.",
"probedAt": "2026-10-23T16:24:29.630Z",
"models": [
{
Expand All @@ -18,8 +18,10 @@
"id": "claude-fable-5-1",
"owner": "anthropic",
"entitled": true,
"chat": "conforms",
"deviations": [],
"chat": "array-content",
"deviations": [
"`content` is an array[2] of {reasoning, text} with keys {summary, text, type}; the spec requires a string"
],
"nativeDialect": "anthropic",
"responses": false,
"webSearch": false,
Expand Down Expand Up @@ -550,8 +552,10 @@
"id": "claude-sonnet-5-5",
"owner": "anthropic",
"entitled": true,
"chat": "conforms",
"deviations": [],
"chat": "array-content",
"deviations": [
"`content` is an array[2] of {reasoning, text} with keys {summary, text, type}; the spec requires a string"
],
"nativeDialect": "anthropic",
"responses": false,
"webSearch": false,
Expand Down
40 changes: 40 additions & 0 deletions src/app/models/content-shape-docs.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import fs from 'fs';
import path from 'path';

import { describe, it, expect } from 'vitest';

import capabilities from './capabilities.json';

// The models page tells readers which models need array handling on chat completions.
// It is prose, so nothing regenerates it when capabilities.json changes.
const modelsDoc = fs.readFileSync(
path.resolve(__dirname, '../../../content/docs/ai-gateway/models.md'),
'utf-8'
);

const admonition = () => {
const start = modelsDoc.indexOf('title="Content shape varies by model"');
const end = modelsDoc.indexOf('</Admonition>', start);
expect(start).toBeGreaterThan(-1);
return modelsDoc.slice(start, end);
};

describe('"Content shape varies by model" in the AI Gateway models doc', () => {
it('names exactly the models measured as array-content', () => {
const servedIds = new Set(capabilities.models.map((m) => m.id));
const named = [...admonition().matchAll(/`([a-z0-9-]+)`/g)]
.map(([, id]) => id)
.filter((id) => servedIds.has(id));
const measured = capabilities.models.filter((m) => m.chat === 'array-content').map((m) => m.id);

expect([...new Set(named)].sort()).toEqual(measured.sort());
});

it('points at the provider and the native endpoints', () => {
const text = admonition();

expect(text).toContain('@neon/ai-sdk-provider');
expect(text).toContain('/docs/ai-gateway/anthropic-messages');
expect(text).toContain('/docs/ai-gateway/gemini');
});
});
2 changes: 1 addition & 1 deletion src/app/models/examples.js
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ print(resp.choices[0].message.content)
});

// cURL swaps to a provider-native dialect only when the unified route is the broken one.
// A native dialect on its own is not a reason to leave `/v1/chat/completions` — the eight
// A native dialect on its own is not a reason to leave `/v1/chat/completions` — the
// conforming Claude models are served at /anthropic/v1/messages too, and showing that
// instead would trade a portable snippet for a vendor-specific one to fix nothing.
const curlChat = (model, nativeDialect, responsesOnly, arrayContent) => {
Expand Down
16 changes: 16 additions & 0 deletions src/app/models/route.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,22 @@ describe('GET /models', () => {
expect(curl.variantReason).toBeTruthy();
});

// Both return a string for a trivial prompt, so only a reasoning probe catches them.
it.each(['claude-sonnet-5-5', 'claude-fable-5-1'])(
'serves no OpenAI SDK chat examples for %s, which returns array content once it reasons',
async (id) => {
const res = await GET(request(`?model=${id}`));
const { model } = await body(res);
const ids = model.examples.map((e) => e.id);
const curl = model.examples.find((e) => e.id === 'curl');

expect(model.capabilities.chat).toBe('array-content');
expect(ids).not.toContain('typescript');
expect(ids).not.toContain('python');
expect(curl.endpoint).toBe('/anthropic/v1/messages');
}
);

it('keeps cURL on chat completions for a conforming Claude model', async () => {
const res = await GET(request('?model=claude-haiku-4-5'));
const { model } = await body(res);
Expand Down
Loading