Add Clef Omni audio and video support

CHANGELOG.md

@@ -8,6 +8,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## langchain-cloudflare
 
+### [0.4.2] - 2026-10-11
+
+#### Added
+
+- **Clef Omni**: `CloudflareWorkersAIDecisionModel` supports
+  `@cf/cloudflare/clef-omni` through synchronous REST, asynchronous REST, and
+  Python Workers AI bindings, preserving native answers, probabilities, and
+  token usage.
+- **Audio and video inputs**: Clef Omni accepts optional `audio` (WAV/MP3) and
+  `videos` (MP4/WebM) lists as embedded data URLs or `content_type`/`base64`
+  objects alongside text, JSON, and images. The `/decision` Worker example
+  forwards these inputs. Older Clef models reject audio and video arguments
+  before calling the provider.
+- **Multimodal integration coverage**: Shared synthetic fixtures test
+  square/circle videos, open/closed speech recordings, video soundtracks, and
+  mixed image/audio/video requests in both encodings through REST and Python
+  Worker bindings.
+
+---
+
 ### [0.4.1] - 2026-10-03
 
 #### Added

libs/langchain-cloudflare/README.md

@@ -71,8 +71,9 @@ its model identity when a batch uses different fallback models.
 ## Decision Models (Clef)
 
 `CloudflareWorkersAIDecisionModel` supports
-[`@cf/cloudflare/clef`](https://developers.cloudflare.com/workers-ai/models/clef/)
-and [`@cf/cloudflare/clef-flash`](https://developers.cloudflare.com/workers-ai/models/clef-flash/).
+[`@cf/cloudflare/clef`](https://developers.cloudflare.com/workers-ai/models/clef/),
+[`@cf/cloudflare/clef-flash`](https://developers.cloudflare.com/workers-ai/models/clef-flash/),
+and [`@cf/cloudflare/clef-omni`](https://developers.cloudflare.com/workers-ai/models/clef-omni/).
 They evaluate text or JSON state against typed questions and return probabilities.
 Question types are `noul` (yes/no probability), `choice` (an option with its
 probability distribution), and `score` (a probability-weighted ordered rubric).
@@ -121,6 +122,44 @@ and `usage` fields. Pass optional `images` as embedded PNG/JPEG/WebP data URLs o
 `{"content_type": "image/png", "base64": "..."}` objects (up to four images).
 The Worker example exposes this operation at `/decision`.
 
+### Clef Omni audio and video
+
+Clef Omni accepts audio and video alongside text, JSON, and images. It returns
+the same decision probabilities and usage as the other Clef models.
+
+```python
+import base64
+from pathlib import Path
+
+from langchain_cloudflare import CloudflareWorkersAIDecisionModel
+
+model = CloudflareWorkersAIDecisionModel(model_name="@cf/cloudflare/clef-omni")
+audio = base64.b64encode(Path("recording.wav").read_bytes()).decode("ascii")
+result = model.evaluate(
+    state="Evaluate the attached recording.",
+    questions={
+        "speech": {
+            "type": "noul",
+            "instructions": "Does the recording contain human speech?",
+        }
+    },
+    audio=[f"data:audio/wav;base64,{audio}"],
+)
+```
+
+Both `evaluate()` and `aevaluate()` accept optional `audio` and `videos` lists.
+Each item is a base64 data URL or a `{"content_type": "...", "base64": "..."}`
+object. Supported formats are WAV/MP3 audio and MP4/WebM video; remote URLs
+are unsupported. Audio and video arguments require Clef Omni. In a Python
+Worker, construct the model with `binding=self.env.AI` and use `aevaluate()`.
+
+- Audio: up to four clips, each at most 8 MiB and 300 seconds.
+- Video: up to two clips, each at most 16 MiB and 60 seconds, sampled at two
+  frames per second. Video soundtracks are used when every supplied video has one.
+- Combined decoded audio and video: at most 16 MiB per request.
+- Media and questions consume the 64,000-token context window. If they exceed
+  it, the request fails; otherwise, text state is truncated to the remaining space.
+
 ## Embeddings
 
 `CloudflareWorkersAIEmbeddings` class exposes embeddings from [CloudflareWorkersAI](https://developers.cloudflare.com/workers-ai/).

libs/langchain-cloudflare/examples/workers/src/entry.py

@@ -1168,6 +1168,8 @@ async def handle_decision(self, request):
             state=data["state"],
             questions=data["questions"],
             images=data.get("images"),
+            audio=data.get("audio"),
+            videos=data.get("videos"),
         )
         return Response.json(result)
 

libs/langchain-cloudflare/langchain_cloudflare/decision_models.py

@@ -1,4 +1,4 @@
-"""Cloudflare Workers AI decision models (Clef and Clef Flash)."""
+"""Cloudflare Workers AI decision models (Clef, Clef Flash, and Clef Omni)."""
 
 # MARK: - IMPORTS
 from typing import Any, Dict, List, Literal, Optional, Union
@@ -16,7 +16,11 @@
 )
 
 # MARK: - MODELS
-DECISION_MODELS = ("@cf/cloudflare/clef", "@cf/cloudflare/clef-flash")
+DECISION_MODELS = (
+    "@cf/cloudflare/clef",
+    "@cf/cloudflare/clef-flash",
+    "@cf/cloudflare/clef-omni",
+)
 
 
 # MARK: - DECISION MODEL
@@ -49,9 +53,9 @@ class CloudflareWorkersAIDecisionModel(BaseModel):
 
     model_config = ConfigDict(extra="forbid", protected_namespaces=())
 
-    model_name: Literal["@cf/cloudflare/clef", "@cf/cloudflare/clef-flash"] = (
-        "@cf/cloudflare/clef"
-    )
+    model_name: Literal[
+        "@cf/cloudflare/clef", "@cf/cloudflare/clef-flash", "@cf/cloudflare/clef-omni"
+    ] = "@cf/cloudflare/clef"
     api_base_url: str = "https://api.cloudflare.com/client/v4/accounts"
     account_id: str = Field(default_factory=from_env("CF_ACCOUNT_ID", default=""))
     api_token: SecretStr = Field(
@@ -91,15 +95,22 @@ def _prepare_payload(
         state: Any,
         questions: Dict[str, Any],
         images: Optional[List[Union[str, Dict[str, str]]]],
+        audio: Optional[List[Union[str, Dict[str, str]]]],
+        videos: Optional[List[Union[str, Dict[str, str]]]],
     ) -> Dict[str, Any]:
         """Derive the required body selector from the endpoint model ID."""
+        if (audio is not None or videos is not None) and self.model_name != (
+            "@cf/cloudflare/clef-omni"
+        ):
+            raise ValueError("Audio and video inputs require @cf/cloudflare/clef-omni")
         payload: Dict[str, Any] = {
             "model": self.model_name.rsplit("/", 1)[-1],
             "state": state,
             "questions": questions,
         }
-        if images is not None:
-            payload["images"] = images
+        for field, media in (("images", images), ("audio", audio), ("videos", videos)):
+            if media is not None:
+                payload[field] = media
         return payload
 
     # MARK: - EVALUATION
@@ -109,16 +120,20 @@ def evaluate(
         state: Any,
         questions: Dict[str, Any],
         images: Optional[List[Union[str, Dict[str, str]]]] = None,
+        audio: Optional[List[Union[str, Dict[str, str]]]] = None,
+        videos: Optional[List[Union[str, Dict[str, str]]]] = None,
     ) -> Dict[str, Any]:
         """Evaluate text or JSON state through REST.
 
         ``questions`` maps 1–64 IDs to native question objects. Optional images
         are embedded PNG/JPEG/WebP data URLs or ``{content_type, base64}``
-        objects; remote URLs are unsupported. Use :meth:`aevaluate` in Workers.
+        objects; remote URLs are unsupported. Clef Omni also accepts ``audio``
+        (WAV/MP3) and ``videos`` (MP4/WebM) in the same embedded formats.
+        Use :meth:`aevaluate` in Workers.
         """
         if self.binding is not None:
             raise ValueError("Use aevaluate() with a Workers AI binding")
-        payload = self._prepare_payload(state, questions, images)
+        payload = self._prepare_payload(state, questions, images, audio, videos)
         response = requests.post(
             url=self._inference_url,
             headers=self.headers,
@@ -134,13 +149,15 @@ async def aevaluate(
         state: Any,
         questions: Dict[str, Any],
         images: Optional[List[Union[str, Dict[str, str]]]] = None,
+        audio: Optional[List[Union[str, Dict[str, str]]]] = None,
+        videos: Optional[List[Union[str, Dict[str, str]]]] = None,
     ) -> Dict[str, Any]:
         """Evaluate the same native input through async REST or ``env.AI.run``.
 
         Returns the same dictionary as :meth:`evaluate`, including each
         question's probabilities, confidence where applicable, and token usage.
         """
-        payload = self._prepare_payload(state, questions, images)
+        payload = self._prepare_payload(state, questions, images, audio, videos)
         if self.binding is not None:
             js_payload = convert_payload_for_binding(payload)
             options = create_binding_run_options(

libs/langchain-cloudflare/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "langchain-cloudflare"
-version = "0.4.1"
+version = "0.4.2"
 description = "Langchain Integrations for Cloudflare's WorkersAI and Vectorize"
 readme = "README.md"
 license = "MIT"

libs/langchain-cloudflare/tests/conftest.py

@@ -16,6 +16,11 @@
 import requests
 from dotenv import load_dotenv
 
+from tests.decision_helpers import (
+    OMNI_MEDIA_CASES,
+    RED_IMAGE,
+    decision_media_case,
+)
 from tests.press_release_examples import PRESS_RELEASE_EXAMPLES
 
 # MARK: - Collection Hooks
@@ -70,10 +75,21 @@ def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:
 
 
 @pytest.fixture(
-    params=["text", "json", "image_data_url", "image_base64", *PRESS_RELEASE_EXAMPLES]
+    params=[
+        "text",
+        "json",
+        "image_data_url",
+        "image_base64",
+        *PRESS_RELEASE_EXAMPLES,
+        *OMNI_MEDIA_CASES,
+    ]
 )
 def decision_case(request):
-    """Native Clef inputs shared by REST and real Worker binding tests."""
+    """Native Clef inputs shared by REST and Worker binding tests."""
+    if request.param in OMNI_MEDIA_CASES:
+        if request.getfixturevalue("model") != "@cf/cloudflare/clef-omni":
+            pytest.skip("Audio and video inputs require Clef Omni")
+        return decision_media_case(request.param)
     payload = {
         "state": "Checkout has been failing for every customer for the last hour.",
         "questions": {
@@ -99,10 +115,7 @@ def decision_case(request):
         payload["state"] = {"service": "checkout", "failing": True, "affected": "all"}
     elif request.param.startswith("image"):
         # Fixed 16x16 solid-red PNG, with no image library dependency.
-        image = (
-            "iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAIAAACQkWg2AAAAF0lEQVR4nGP4z8BA"
-            "EiJN9aiGUQ1DSgMAkPn/Afnh+ngAAAAASUVORK5CYII="
-        )
+        image = RED_IMAGE
         payload = {
             "state": "Evaluate the attached image.",
             "questions": {

libs/langchain-cloudflare/tests/decision_helpers.py

@@ -1,8 +1,93 @@
-"""Shared assertions for real REST and Worker decision-model results."""
+"""Shared fixtures and assertions for REST and Worker decision-model results."""
 
 # MARK: - IMPORTS
+import base64
+from pathlib import Path
+
 import pytest
 
+# MARK: - MEDIA FIXTURES
+RED_IMAGE = (
+    "iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAIAAACQkWg2AAAAF0lEQVR4nGP4z8BA"
+    "EiJN9aiGUQ1DSgMAkPn/Afnh+ngAAAAASUVORK5CYII="
+)
+OMNI_MEDIA_CASES = [
+    f"omni_{case}-{encoding}"
+    for case in (
+        "audio_open",
+        "audio_closed",
+        "video_square",
+        "video_circle",
+        "video_sound_square_open",
+        "video_sound_circle_closed",
+        "mixed",
+    )
+    for encoding in ("data_url", "base64")
+]
+
+
+def decision_media_case(case_id: str) -> tuple[dict, dict]:
+    """Attach fixed media without putting its expected answers in the state."""
+    case, encoding = case_id.removeprefix("omni_").split("-")
+    media_dir = Path(__file__).parent / "fixtures" / "clef_omni"
+
+    def embedded(content_type: str, encoded: str):
+        if encoding == "data_url":
+            return f"data:{content_type};base64,{encoded}"
+        return {"content_type": content_type, "base64": encoded}
+
+    def media(filename: str, content_type: str):
+        encoded = base64.b64encode((media_dir / filename).read_bytes()).decode("ascii")
+        return embedded(content_type, encoded)
+
+    speech = {
+        "type": "choice",
+        "instructions": "Which statement is spoken in the audio recording?",
+        "criteria": {"open": "The door is open.", "closed": "The door is closed."},
+    }
+    shape = {
+        "type": "choice",
+        "instructions": "Which geometric shape is shown in the video?",
+        "criteria": {
+            "square": "A square",
+            "circle": "A circle",
+            "triangle": "A triangle",
+        },
+    }
+    payload = {"state": "Evaluate the attached media.", "questions": {}}
+    expected = {}
+    if case.startswith("audio_"):
+        spoken = case.removeprefix("audio_")
+        payload["audio"] = [media(f"door_{spoken}.wav", "audio/wav")]
+        payload["questions"]["speech"] = speech
+        expected["speech"] = spoken
+    elif case.startswith("video_"):
+        clip = case.removeprefix("video_").removeprefix("sound_")
+        payload["videos"] = [media(f"{clip}.mp4", "video/mp4")]
+        payload["questions"]["shape"] = shape
+        expected["shape"] = clip.split("_")[0]
+        if case.startswith("video_sound_"):
+            speech["instructions"] = (
+                "Which statement is spoken in the video's soundtrack?"
+            )
+            payload["questions"]["speech"] = speech
+            expected["speech"] = clip.split("_")[1]
+    else:
+        payload["images"] = [embedded("image/png", RED_IMAGE)]
+        payload["audio"] = [media("door_open.wav", "audio/wav")]
+        payload["videos"] = [media("circle.mp4", "video/mp4")]
+        payload["questions"] = {
+            "speech": speech,
+            "shape": shape,
+            "color": {
+                "type": "choice",
+                "instructions": "What is the dominant color of the attached image?",
+                "criteria": {"red": "Red", "green": "Green", "blue": "Blue"},
+            },
+        }
+        expected = {"speech": "open", "shape": "circle", "color": "red"}
+    return payload, expected
+
 
 # MARK: - ASSERTIONS
 def assert_decision_result(
@@ -27,7 +112,11 @@ def assert_decision_result(
         if question["type"] == "choice":
             assert set(probabilities) == set(question["criteria"])
             assert answer["choice"] == max(probabilities, key=probabilities.get)
-            assert answer["choice"] == expected[key]
+            message = (
+                f"{key}: expected {expected[key]!r}, got {answer['choice']!r}; "
+                f"probabilities={probabilities}"
+            )
+            assert answer["choice"] == expected[key], message
             if key == "context" and answer["choice"] != "none":
                 context = question["criteria"][answer["choice"]]
                 assert context in payload["state"]["press_release"]

libs/langchain-cloudflare/tests/fixtures/clef_omni/README.md

@@ -0,0 +1,34 @@
+# Clef Omni media fixtures
+
+These synthetic fixtures contain only the statements and animations below.
+Tests read the committed bytes; speech synthesis and FFmpeg are not test dependencies.
+
+| File | Content |
+| --- | --- |
+| `door_open.wav` | “The door is open.” |
+| `door_closed.wav` | “The door is closed.” |
+| `square.mp4` | A red square on a white background; no audio. |
+| `circle.mp4` | A red circle on a white background; no audio. |
+| `square_open.mp4` | The square animation with the open-door recording. |
+| `circle_closed.mp4` | The circle animation with the closed-door recording. |
+
+Audio was synthesized with macOS `say`, the Samantha voice, at 140 words per
+minute, as mono 16-bit PCM WAV at 16 kHz. Video contains 30 RGB frames at
+160×120 pixels and 10 fps, encoded with FFmpeg's `libx264` encoder, CRF 24,
+and `yuv420p` pixel format. The square is 60×60 pixels at y=30, with its
+left edge moving from x=95 to x=5. The circle has radius 30 pixels and a
+center moving from (34.5, 59.5) to (124.5, 59.5). Each animation holds its
+starting position for the first ten frames, moves linearly over the middle
+ten frames, and holds its final position for the last ten frames. Soundtracks
+are AAC at 32 kbit/s, padded to the three-second video length. Container
+metadata was stripped with `-map_metadata -1`.
+
+The same media bytes are tested as data URLs and as `content_type`/`base64`
+objects through synchronous REST, asynchronous REST, and Python Worker bindings.
+The mixed-media case uses the existing red PNG fixture, the open-door audio,
+and the silent circle video. Prompts do not reveal the expected answers.
+
+Acceptance checks distinguish square from circle and open-door speech from
+closed-door speech, including video soundtracks. They require correct native
+answers, valid probability distributions, and token usage. Motion-direction
+accuracy is a separate exploratory model-quality check.

libs/langchain-cloudflare/tests/integration_tests/test_workersai_models.py

@@ -735,7 +735,7 @@ def test_tool_strategy_json_schema_invoke(
 
 
 class TestDecisionModels:
-    """Clef's native decision contract through synchronous and async REST."""
+    """Clef decisions, including Omni audio/video, through sync and async REST."""
 
     @pytest.mark.parametrize("model", DECISION_MODELS)
     @pytest.mark.parametrize(