Skip to content
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/ocr.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
| Cost Tracking | ✅ |
| Logging | ✅ (Basic Logging not supported) |
| Load Balancing | ✅ |
| Supported Providers | `mistral`, `azure_ai`, `vertex_ai` |
| Supported Providers | `mistral`, `azure_ai`, `vertex_ai`, `cohere` |

:::tip

Expand Down Expand Up @@ -345,6 +345,7 @@ The response follows Mistral's OCR format with the following structure:
| Provider | Link to Usage |
|-------------|--------------------|
| Mistral AI | [Usage](#quick-start) |
| Azure AI | [Usage](../docs/providers/azure_ocr) |
| Azure AI (Mistral, Cohere Parse) | [Usage](../docs/providers/azure_ocr) |
| Vertex AI | [Usage](../docs/providers/vertex_ocr) |
| Cohere Parse | [Usage](../docs/providers/cohere#parse-ocr) |

66 changes: 63 additions & 3 deletions docs/providers/azure_ocr.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# Azure AI OCR (Mistral)
# Azure AI OCR (Mistral, Cohere Parse)

## Overview

| Property | Details |
|-------|-------|
| Description | Azure AI OCR provides document intelligence capabilities powered by Mistral, enabling text extraction from PDFs and images |
| Description | Azure AI OCR provides document intelligence capabilities powered by Mistral and Cohere Parse, enabling text extraction from PDFs and images |
| Provider Route on LiteLLM | `azure_ai/` |
| Supported Operations | `/ocr` |
| Link to Provider Doc | [Azure AI ↗](https://ai.azure.com/)

Extract text from documents and images using Azure AI's OCR models, powered by Mistral.
Extract text from documents and images using Azure AI's OCR models, powered by Mistral. Cohere Parse deployments are covered [below](#cohere-parse).

## Quick Start

Expand Down Expand Up @@ -146,9 +146,69 @@ response = await litellm.aocr(
Azure AI OCR endpoints don't have internet access. LiteLLM automatically converts public URLs to base64 data URIs before sending requests to Azure AI.
:::

## Cohere Parse

Azure AI Foundry also serves [Cohere Parse](https://ai.azure.com/catalog/models/Cohere-parse-v5) through the same `/ocr` endpoint. Use `azure_ai/<deployment name>`: a deployment whose name contains both `cohere` and `parse` (the catalog's default name `Cohere-parse-v5` does) is sent to the Cohere Parse API on your Foundry resource, at `{api_base}/providers/cohere/v2/parse`. Other names keep routing to Mistral OCR, so keep `cohere` and `parse` in the deployment name if you rename it.

Parse accepts `image_url` documents only, an image URL or a base64 `data:image/...` URI. PDFs and `document_url` inputs are rejected with a 400 before anything is sent to Azure. Foundry cannot fetch external URLs, so LiteLLM downloads a remote image and sends it inline as a data URI, the same conversion it applies for the Mistral models above.

### **LiteLLM SDK**

```python showLineNumbers title="Cohere Parse on Azure AI"
import litellm
import os

os.environ["AZURE_AI_API_KEY"] = ""
os.environ["AZURE_AI_API_BASE"] = "https://<resource>.services.ai.azure.com"

response = litellm.ocr(
model="azure_ai/Cohere-parse-v5",
document={
"type": "image_url",
"image_url": "https://raw.githubusercontent.com/mistralai/cookbook/refs/heads/main/mistral/ocr/receipt.png",
},
output_format="markdown",
)

for page in response.pages:
print(page.markdown)
print(response.usage_info.pages_processed)
```

### **LiteLLM PROXY**

```yaml showLineNumbers title="proxy_config.yaml"
model_list:
- model_name: azure-cohere-parse
litellm_params:
model: azure_ai/Cohere-parse-v5
api_key: "os.environ/AZURE_AI_API_KEY"
api_base: "os.environ/AZURE_AI_API_BASE"
```

```bash showLineNumbers title="Test request"
curl http://0.0.0.0:4000/v1/ocr \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "azure-cohere-parse",
"document": {
"type": "image_url",
"image_url": "https://raw.githubusercontent.com/mistralai/cookbook/refs/heads/main/mistral/ocr/receipt.png"
}
}'
```

`output_format` accepts `markdown` (default) or `blocks`, and `req_format: native` returns Cohere's own response body instead of the LiteLLM OCR shape. Cost tracking bills `usage_info.pages_processed` at the per-page price in the model cost map.

The model cost map prices `azure_ai/Cohere-parse-v5` at Cohere's published rate of $1.50 per 1,000 pages, the price the Foundry catalog links to for this model.

Health checks (`/health` and the Admin UI's Test Connection button) send Parse a small PNG instead of the PDF used for Mistral OCR. The `ocr` probe is picked from the model cost map for `Cohere-parse-v5`; a deployment under any other name needs `model_info: {mode: ocr}` in its `model_list` entry, like every other non-chat model.

## Supported Models

- `mistral-document-ai-2505` - Latest Mistral OCR model on Azure AI
- `Cohere-parse-v5` - Cohere Parse, image documents only

Use the Azure AI provider prefix: `azure_ai/<model-name>`

79 changes: 78 additions & 1 deletion docs/providers/cohere.md
Original file line number Diff line number Diff line change
Expand Up @@ -348,4 +348,81 @@ curl http://0.0.0.0:4000/rerank \
```

</TabItem>
</Tabs>
</Tabs>

## Parse (OCR)

[Cohere Parse](https://docs.cohere.com/reference/parse) turns a document image into markdown. LiteLLM serves it through the [`/ocr` endpoint](../ocr), so requests and responses use the same shape as every other OCR provider and each call is cost tracked per billed page.

Parse accepts `image_url` documents only: an image URL or a base64 `data:image/...` URI. PDFs and `document_url` inputs are rejected with a 400 before anything is sent to Cohere.

<Tabs>
<TabItem value="sdk" label="LiteLLM SDK Usage">

```python showLineNumbers
import os
from litellm import ocr

os.environ["COHERE_API_KEY"] = ""

response = ocr(
model="cohere/parse-v5.0",
document={
"type": "image_url",
"image_url": "https://raw.githubusercontent.com/mistralai/cookbook/refs/heads/main/mistral/ocr/receipt.png",
},
)

for page in response.pages:
print(page.markdown)
print(response.usage_info.pages_processed)
```
</TabItem>

<TabItem value="proxy" label="LiteLLM Proxy Usage">

**Setup**

Add this to your litellm proxy config.yaml

```yaml
model_list:
- model_name: cohere-parse
litellm_params:
model: cohere/parse-v5.0
api_key: os.environ/COHERE_API_KEY
```

Start litellm

```bash
litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
```

Test request

```bash
curl http://0.0.0.0:4000/v1/ocr \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere-parse",
"document": {
"type": "image_url",
"image_url": "https://raw.githubusercontent.com/mistralai/cookbook/refs/heads/main/mistral/ocr/receipt.png"
}
}'
```
</TabItem>
</Tabs>

### Supported Parameters

| Parameter | Values | Description |
|-----------|--------|-------------|
| `output_format` | `markdown` (default), `blocks` | Cohere's output layout. With `blocks`, each page carries Cohere's `blocks` array and `markdown` is empty |
| `req_format` | `litellm` (default), `native` | `native` returns Cohere's own response body instead of the LiteLLM OCR shape |

Cohere Parse deployed on Azure AI Foundry is covered in [Azure AI OCR](./azure_ocr#cohere-parse).
Loading