docs: explain the default prompt template on the ollama/ provider - #1191
docs: explain the default prompt template on the ollama/ provider#1191mdevhurt1 wants to merge 2 commits into
Conversation
The page recommends ollama_chat "for better responses" without saying why. The reason is that ollama/ posts to /api/generate, so litellm flattens the messages array into one text prompt, and with no template registered it uses a hard-coded "### System:" / "### User:" / "### Assistant:" template. Those markers replace the model's own chat template, and some models echo them back as response content. Document the default, why ollama_chat/ is the recommendation, and register_prompt_template() as the workaround for callers that need /api/generate, in both the SDK and the proxy config.yaml form. Refs BerriAI/litellm#39724
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
🟡 Changes recommended
The new section contains a few documentation inaccuracies/inconsistencies (notably around required proxy prompt-template fields and the shown default prompt example) that should be corrected before merge.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the Ollama provider documentation to explain why ollama_chat/ is recommended for chat models, focusing on how the ollama/ completion prefix flattens chat messages into a single prompt and can apply a default prompt template that does not match the model’s native chat formatting.
Changes:
- Expands the existing recommendation to use
ollama_chat/by explaining the behavioral difference between/api/generateand/api/chat. - Adds a new “Prompt templates” section showing the default flattened prompt shape and how to avoid it via
ollama_chat/or custom prompt templates (SDK and proxy config examples).
File summaries
| File | Description |
|---|---|
| docs/providers/ollama.md | Adds a new section documenting the default prompt-template behavior for ollama/, plus remediation options and links to existing prompt-template docs. |
Review details
Suppressed comments (2)
docs/providers/ollama.md:300
- The sentence "All three of
roles,initial_prompt_valueandfinal_prompt_valuehave to be set" conflicts with the prompt-template spec elsewhere in the docs, whereinitial_prompt_valueandfinal_prompt_valueare described as optional. Consider rephrasing to indicate this is recommended for a clean override, not strictly required.
On the proxy, set the same template in your `config.yaml` under `litellm_params`. All three of `roles`, `initial_prompt_value` and `final_prompt_value` have to be set:
docs/providers/ollama.md:268
- This section says the default template includes
### Assistant:, but the example prompt block never shows the assistant marker. Including it makes it clearer what text is actually sent to/api/generateand where the model starts generating.
`ollama/` is the completion provider. It posts to `/api/generate`, which takes a single text prompt rather than a messages array, so LiteLLM flattens your `messages` into one string before sending it. When no prompt template is registered for the model, it uses a hard-coded `### System:` / `### User:` / `### Assistant:` template:
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| :::info | ||
|
|
||
| We recommend using [ollama_chat](#using-ollama-apichat) for better responses. | ||
| We recommend using [ollama_chat](#using-ollama-apichat) for chat models. The `ollama/` prefix flattens your messages into a single text prompt and replaces the model's own chat template, which usually lowers response quality. See [Prompt templates and the `ollama/` prefix](#prompt-templates). |
There was a problem hiding this comment.
Added the slash and code formatting to the line 13 link text in 598fd46. Lines 268 and 300 carry no ollama_chat reference to change
|
Two notes on the suppressed Copilot comments. The "all three keys" sentence describes released proxy behavior: today the ollama path raises a KeyError when either prompt value is missing, and BerriAI/litellm#39762 is open to make the two optional, so this page gets updated once that lands in a release. The example prompt block is byte-accurate: the flattener only emits "### Assistant:" when the conversation contains an assistant message, and it appends no trailing assistant marker, which is part of the problem the section describes |
What this changes
The Ollama page has recommended
ollama_chat"for better responses" since 2024 without ever saying why. This adds a section that explains the reason.ollama/is the completion provider, so it posts to/api/generate, which takes a single text prompt rather than a messages array. LiteLLM flattens the messages, and with no template registered for the model it uses a hard-coded### System:/### User:/### Assistant:template. That replaces the chat template stored in the model, the one Ollama would apply on/api/chat, and some models echo the markers back as response content. The request still returns 200, so it is easy to miss.The new section documents the default, shows what the flattened prompt looks like, and gives the two ways out: switch to
ollama_chat/, or register your own template. The register path is shown in both the SDK form and the proxyconfig.yamlform, with a link to the existing "Set Custom Prompt Templates" section.Nothing else on the site said any of this.
ollama_ptand the### System:markers had zero hits across the whole docs repo before this change.Verification
npm run lint:writingpasses with no prose warnings on the page, andnpm run buildexits 0 withonBrokenLinksandonBrokenAnchorsset to throw, so both new links are checked. The new#prompt-templatesanchor and the existing#using-ollama-apichatanchor are both present in the rendered HTML.Relevant issues
Refs BerriAI/litellm#39724
Note for reviewer: safe to merge in either order relative to BerriAI/litellm PR (bugfix/ollama-default-template-warning). This page describes only released behavior.