Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/providers/azure_ai.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,7 @@ LiteLLM supports **ALL** azure ai models. Here's a few examples:

| Model Name | Function Call |
|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Claude Opus 5 | `completion(model="azure_ai/claude-opus-5", messages)` |
| Cohere command-r-plus | `completion(model="azure_ai/command-r-plus", messages)` |
| Cohere command-r | `completion(model="azure_ai/command-r", messages)` |
| mistral-large-latest | `completion(model="azure_ai/mistral-large-latest", messages)` |
Expand Down Expand Up @@ -354,7 +355,7 @@ export AZURE_AI_API_BASE="https://my-resource.services.ai.azure.com/anthropic"

```yaml
model_list:
- model_name: claude-4-azure
- model_name: azure-claude
litellm_params:
model: azure_ai/{{anthropic_large}}
api_key: os.environ/AZURE_AI_API_KEY
Expand All @@ -374,11 +375,11 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
"model": "claude-4-azure",
"model": "azure-claude",
"messages": [
{
"role": "user",
"content": "How do I use Claude Opus 4 via Azure Anthropic in LiteLLM?"
"content": "How do I use Claude Opus via Azure Anthropic in LiteLLM?"
}
],
"max_tokens": 1024
Expand Down
12 changes: 7 additions & 5 deletions docs/providers/bedrock.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ response = completion(
<TabItem value="proxy" label="PROXY">
```yaml
model_list:
- model_name: bedrock-claude-sonnet-4-5
- model_name: bedrock-claude-sonnet-5
litellm_params:
model: bedrock/us.anthropic.{{anthropic}}
api_key: os.environ/AWS_BEARER_TOKEN_BEDROCK
Expand Down Expand Up @@ -93,7 +93,7 @@ Here's how to call Bedrock with the LiteLLM Proxy Server

```yaml
model_list:
- model_name: bedrock-claude-sonnet-4-5
- model_name: bedrock-claude-sonnet-5
litellm_params:
model: bedrock/us.anthropic.{{anthropic}}
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
Expand Down Expand Up @@ -2020,13 +2020,15 @@ LiteLLM supports ALL Bedrock models.

Here's an example of using a bedrock model with LiteLLM. For a complete list, refer to the [model cost map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)

| Model Name | Command |
|----------------------------|------------------------------------------------------------------|
| Model Name | Command | Required OS Variables |
|------------|---------|-----------------------|
| Anthropic Claude Opus 5 | `completion(model='bedrock/us.anthropic.claude-opus-5', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']` |
| Amazon Nova 2 Lite | `completion(model='bedrock/amazon.nova-2-lite-v1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']`, `os.environ['AWS_REGION_NAME']` |
| GPT-OSS 20B | `completion(model='bedrock/converse/openai.gpt-oss-20b-1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']`, `os.environ['AWS_REGION_NAME']` |
| GPT-OSS 120B | `completion(model='bedrock/converse/openai.gpt-oss-120b-1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']`, `os.environ['AWS_REGION_NAME']` |
| Deepseek R1 | `completion(model='bedrock/us.deepseek.r1-v1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']` |
| Anthropic Claude Sonnet 4.5 | `completion(model='bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']` |
| Anthropic Claude-V3.5 Sonnet | `completion(model='bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']` |
| Anthropic Claude Haiku 4.5 | `completion(model='bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']` |
| Anthropic Claude-V3 sonnet | `completion(model='bedrock/anthropic.claude-3-sonnet-20240229-v1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']` |
| Anthropic Claude-V3 Haiku | `completion(model='bedrock/anthropic.claude-3-haiku-20240307-v1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']` |
| Anthropic Claude-V3 Opus | `completion(model='bedrock/anthropic.claude-3-opus-20240229-v1:0', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']` |
Expand Down
10 changes: 4 additions & 6 deletions docs/providers/cerebras.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,19 +23,18 @@ import os

os.environ['CEREBRAS_API_KEY'] = ""
response = completion(
model="cerebras/llama3-70b-instruct",
model="cerebras/gpt-oss-120b",
messages=[
{
"role": "user",
"content": "What's the weather like in Boston today in Fahrenheit? (Write in JSON)",
}
],
max_tokens=10,
max_tokens=4096,

# The prompt should include JSON if 'json_object' is selected; otherwise, you will get error code 400.
response_format={ "type": "json_object" },
seed=123,
stop=["\n\n"],
temperature=0.2,
top_p=0.9,
tool_choice="auto",
Expand All @@ -52,20 +51,19 @@ import os

os.environ['CEREBRAS_API_KEY'] = ""
response = completion(
model="cerebras/llama3-70b-instruct",
model="cerebras/gpt-oss-120b",
messages=[
{
"role": "user",
"content": "What's the weather like in Boston today in Fahrenheit? (Write in JSON)",
}
],
stream=True,
max_tokens=10,
max_tokens=4096,

# The prompt should include JSON if 'json_object' is selected; otherwise, you will get error code 400.
response_format={ "type": "json_object" },
seed=123,
stop=["\n\n"],
temperature=0.2,
top_p=0.9,
tool_choice="auto",
Expand Down
28 changes: 14 additions & 14 deletions docs/providers/cohere.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ os.environ["COHERE_API_KEY"] = "cohere key"

# cohere v2 call
response = completion(
model="cohere_chat/command-a-03-2025",
model="cohere_chat/command-a-plus-05-2026",
messages = [{ "content": "Hello, how are you?","role": "user"}]
)
```
Expand All @@ -42,7 +42,7 @@ os.environ["COHERE_API_KEY"] = "cohere key"

# cohere v1 call
response = completion(
model="cohere_chat/v1/command-a-03-2025",
model="cohere_chat/v1/command-a-plus-05-2026",
messages = [{ "content": "Hello, how are you?","role": "user"}]
)
```
Expand All @@ -59,7 +59,7 @@ os.environ["COHERE_API_KEY"] = "cohere key"

# cohere v2 streaming
response = completion(
model="cohere_chat/command-a-03-2025",
model="cohere_chat/command-a-plus-05-2026",
messages = [{ "content": "Hello, how are you?","role": "user"}],
stream=True
)
Expand All @@ -79,7 +79,7 @@ os.environ["COHERE_API_KEY"] = "cohere key"

# cohere v1 streaming
response = completion(
model="cohere_chat/v1/command-a-03-2025",
model="cohere_chat/v1/command-a-plus-05-2026",
messages = [{ "content": "Hello, how are you?","role": "user"}],
stream=True
)
Expand All @@ -106,18 +106,18 @@ Define the cohere models you want to use in the config.yaml
**For Cohere v1 models:**
```yaml showLineNumbers
model_list:
- model_name: command-a-03-2025
- model_name: command-a-plus-05-2026
litellm_params:
model: cohere_chat/v1/command-a-03-2025
model: cohere_chat/v1/command-a-plus-05-2026
api_key: "os.environ/COHERE_API_KEY"
```

**For Cohere v2 models:**
```yaml showLineNumbers
model_list:
- model_name: command-a-03-2025-v2
- model_name: command-a-plus-05-2026-v2
litellm_params:
model: cohere_chat/command-a-03-2025
model: cohere_chat/command-a-plus-05-2026
api_key: "os.environ/COHERE_API_KEY"
```

Expand All @@ -136,7 +136,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <your-litellm-api-key>' \
--data ' {
"model": "command-a-03-2025",
"model": "command-a-plus-05-2026",
"messages": [
{
"role": "user",
Expand All @@ -154,7 +154,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <your-litellm-api-key>' \
--data ' {
"model": "command-a-03-2025-v2",
"model": "command-a-plus-05-2026-v2",
"messages": [
{
"role": "user",
Expand All @@ -175,7 +175,7 @@ client = openai.OpenAI(
)

# request sent to cohere v1 model
response = client.chat.completions.create(model="command-a-03-2025", messages = [
response = client.chat.completions.create(model="command-a-plus-05-2026", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
Expand All @@ -195,7 +195,7 @@ client = openai.OpenAI(
)

# request sent to cohere v2 model
response = client.chat.completions.create(model="command-a-03-2025-v2", messages = [
response = client.chat.completions.create(model="command-a-plus-05-2026-v2", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
Expand All @@ -211,7 +211,7 @@ print(response)
## Supported Models
| Model Name | Function Call |
|------------|----------------|
| command-a-03-2025 | `litellm.completion('command-a-03-2025', messages)` |
| command-a-plus-05-2026 | `litellm.completion('cohere_chat/command-a-plus-05-2026', messages)` |
| command-r-plus-08-2024 | `litellm.completion('command-r-plus-08-2024', messages)` |
| command-r-08-2024 | `litellm.completion('command-r-08-2024', messages)` |
| command-r-plus | `litellm.completion('command-r-plus', messages)` |
Expand Down Expand Up @@ -348,4 +348,4 @@ curl http://0.0.0.0:4000/rerank \
```

</TabItem>
</Tabs>
</Tabs>
22 changes: 10 additions & 12 deletions docs/providers/deepinfra.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import os

os.environ['DEEPINFRA_API_KEY'] = ""
response = completion(
model="deepinfra/meta-llama/Llama-2-70b-chat-hf",
model="deepinfra/moonshotai/Kimi-K3",
messages=[{"role": "user", "content": "write code for saying hi from LiteLLM"}]
)
```
Expand All @@ -41,7 +41,7 @@ import os

os.environ['DEEPINFRA_API_KEY'] = ""
response = completion(
model="deepinfra/meta-llama/Llama-2-70b-chat-hf",
model="deepinfra/moonshotai/Kimi-K3",
messages=[{"role": "user", "content": "write code for saying hi from LiteLLM"}],
stream=True
)
Expand All @@ -51,16 +51,14 @@ for chunk in response:
```

## Chat Models
| Model Name | Function Call |
|------------------|--------------------------------------|
| meta-llama/Meta-Llama-3-8B-Instruct | `completion(model="deepinfra/meta-llama/Meta-Llama-3-8B-Instruct", messages)` |
| meta-llama/Meta-Llama-3-70B-Instruct | `completion(model="deepinfra/meta-llama/Meta-Llama-3-70B-Instruct", messages)` |
| meta-llama/Llama-2-70b-chat-hf | `completion(model="deepinfra/meta-llama/Llama-2-70b-chat-hf", messages)` |
| meta-llama/Llama-2-7b-chat-hf | `completion(model="deepinfra/meta-llama/Llama-2-7b-chat-hf", messages)` |
| meta-llama/Llama-2-13b-chat-hf | `completion(model="deepinfra/meta-llama/Llama-2-13b-chat-hf", messages)` |
| codellama/CodeLlama-34b-Instruct-hf | `completion(model="deepinfra/codellama/CodeLlama-34b-Instruct-hf", messages)` |
| mistralai/Mistral-7B-Instruct-v0.1 | `completion(model="deepinfra/mistralai/Mistral-7B-Instruct-v0.1", messages)` |
| jondurbin/airoboros-l2-70b-gpt4-1.4.1 | `completion(model="deepinfra/jondurbin/airoboros-l2-70b-gpt4-1.4.1", messages)` |

Use `deepinfra/<model-id>` with a model from the [DeepInfra catalog](https://deepinfra.com/models). These are representative chat models:

| Model | LiteLLM model ID |
|-------|------------------|
| GLM-5.3 | `deepinfra/zai-org/GLM-5.3` |
| Kimi K3 | `deepinfra/moonshotai/Kimi-K3` |
| GPT OSS 120B | `deepinfra/openai/gpt-oss-120b` |

## Rerank Endpoint

Expand Down
8 changes: 4 additions & 4 deletions docs/providers/deepseek.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import os

os.environ['DEEPSEEK_API_KEY'] = ""
response = completion(
model="deepseek/deepseek-chat",
model="deepseek/deepseek-v4-pro",
messages=[
{"role": "user", "content": "hello from litellm"}
],
Expand All @@ -34,7 +34,7 @@ import os

os.environ['DEEPSEEK_API_KEY'] = ""
response = completion(
model="deepseek/deepseek-chat",
model="deepseek/deepseek-v4-pro",
messages=[
{"role": "user", "content": "hello from litellm"}
],
Expand All @@ -51,7 +51,7 @@ We support ALL Deepseek models, just set `deepseek/` as a prefix when sending co

| Model Name | Function Call |
|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| deepseek-chat | `completion(model="deepseek/deepseek-chat", messages)` |
| deepseek-v4-pro | `completion(model="deepseek/deepseek-v4-pro", messages)` |
| deepseek-coder | `completion(model="deepseek/deepseek-coder", messages)` |


Expand Down Expand Up @@ -170,4 +170,4 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \

</TabItem>

</Tabs>
</Tabs>
20 changes: 10 additions & 10 deletions docs/providers/fireworks_ai.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,15 @@ import os

os.environ['FIREWORKS_AI_API_KEY'] = ""
response = completion(
model="fireworks_ai/glm-5p2",
model="fireworks_ai/glm-5p3-flash",
messages=[
{"role": "user", "content": "hello from litellm"}
],
)
print(response)
```

A bare serverless slug like `glm-5p2` is expanded to `accounts/fireworks/models/glm-5p2` for you, so you can pass either the short slug or the full resource id.
A bare serverless slug like `glm-5p3-flash` is expanded to `accounts/fireworks/models/glm-5p3-flash` for you, so you can pass either the short slug or the full resource id.

## Sample Usage - Serverless Models - Streaming
```python
Expand All @@ -55,7 +55,7 @@ import os

os.environ['FIREWORKS_AI_API_KEY'] = ""
response = completion(
model="fireworks_ai/glm-5p2",
model="fireworks_ai/glm-5p3-flash",
messages=[
{"role": "user", "content": "hello from litellm"}
],
Expand Down Expand Up @@ -126,9 +126,9 @@ The full resource id (`fireworks_ai/accounts/fireworks/routers/glm-latest`) is s

```yaml
model_list:
- model_name: fireworks-glm-5p2
- model_name: fireworks-glm-5p3-flash
litellm_params:
model: fireworks_ai/glm-5p2
model: fireworks_ai/glm-5p3-flash
api_key: "os.environ/FIREWORKS_AI_API_KEY"
```

Expand All @@ -148,7 +148,7 @@ litellm --config config.yaml
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--data ' {
"model": "fireworks-glm-5p2",
"model": "fireworks-glm-5p3-flash",
"messages": [
{
"role": "user",
Expand All @@ -169,7 +169,7 @@ client = openai.OpenAI(
)

# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="fireworks-glm-5p2", messages = [
response = client.chat.completions.create(model="fireworks-glm-5p3-flash", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
Expand All @@ -193,7 +193,7 @@ from langchain.schema import HumanMessage, SystemMessage

chat = ChatOpenAI(
openai_api_base="http://0.0.0.0:4000", # set openai_api_base to the LiteLLM Proxy
model = "fireworks-glm-5p2",
model = "fireworks-glm-5p3-flash",
temperature=0.1
)

Expand Down Expand Up @@ -449,7 +449,7 @@ We support ALL Fireworks AI models, just set `fireworks_ai/` as a prefix when se

| Model Name | Function Call |
|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| glm-5p2 | `completion(model="fireworks_ai/glm-5p2", messages)` |
| glm-5p3-flash | `completion(model="fireworks_ai/glm-5p3-flash", messages)` |
| deepseek-v4-pro | `completion(model="fireworks_ai/deepseek-v4-pro", messages)` |
| kimi-k3 | `completion(model="fireworks_ai/kimi-k3", messages)` |
| qwen3p8-max | `completion(model="fireworks_ai/qwen3p8-max", messages)` |
Expand Down Expand Up @@ -611,4 +611,4 @@ curl http://0.0.0.0:4000/rerank \

| Model Name | Function Call |
|------------|---------------|
| fireworks/qwen3-reranker-8b | `rerank(model="fireworks_ai/fireworks/qwen3-reranker-8b", query=query, documents=documents)` |
| fireworks/qwen3-reranker-8b | `rerank(model="fireworks_ai/fireworks/qwen3-reranker-8b", query=query, documents=documents)` |
Loading