Add RotorQuant KV cache backend with deferred prefill on Metal

README.md

@@ -20,11 +20,90 @@ a more modern dashboard, richer API workflows, sophisticated cache quantization,
 - Form a small cluster of Macs and split larger models across them.
 - Use a central model store so the cluster downloads once and stages locally.
 - Talk to the cluster through OpenAI Chat Completions, OpenAI Responses, Claude Messages, or Ollama-compatible APIs.
+- Push KV cache memory down hard with rotation-based 3-bit quantization (RotorQuant, OptiQ, TurboQuant) and pick a backend per workload from the dashboard.
+- Use a model-aware reasoning contract that handles toggleable and non-toggleable thinking models without baking assumptions into client code.
 - Experiment with advanced placement modes, RDMA, and KV cache backends when you are ready.
 - Run non-chat workloads such as embeddings and other specialized model flows.
 - Build TTS-oriented and other API-driven workflows on top of the cluster.
 - Actually use your cluster for real inference workloads instead of treating it as a demo.
 
+## Everything Different About Skulk
+
+This is the running list of where Skulk diverges from upstream [exo](https://github.com/exo-explore/exo). It is a living section — every meaningful change should land here when it ships, so anyone evaluating Skulk can see the surface area at a glance.
+
+### Inference and KV cache
+
+- **RotorQuant KV cache backend** — pure-MLX port of IsoQuant 3-bit (block-diagonal quaternion rotations + Lloyd-Max centroids) with **deferred prefill on Metal**, a contribution that does not exist in any upstream project (the llama.cpp fork ships it CUDA-only). GQA-native; no fallback for grouped-query models. See [docs/kv-cache-backends.md](docs/kv-cache-backends.md).
+- **TurboQuant native and adaptive backends** — randomized Hadamard rotation + Lloyd-Max centroids, with an adaptive variant that keeps edge attention layers in fp16 for accuracy.
+- **OptiQ KV cache integration** — wraps `mlx-optiq`'s rotated-space attention path so the rotation cost stays out of the per-token loop on supported (non-GQA) models.
+- **OptiQ mixed-precision weight quantization pipeline** — async wrapper around `mlx-optiq`'s sensitivity analysis and KL-divergence per-layer bit allocation, exposed as a model-store optimization job.
+- **KV prefix cache with snapshot/restore** — LRU-evicted prompt-prefix cache that snapshots SSM and rotating-window cache states so prefix matches are reusable across conversation turns even for hybrid Mamba/Transformer architectures.
+- **Pipeline-parallel prefill for short prompts** — pipelined models now route every prefill through the pipeline path, fixing prior warmup hangs on Gemma-class models.
+- **Force-sequential fallback** — quantized backends transparently fall back to a sequential generator when batch/history mode is incompatible with their cache layout.
+
+### Model capability system
+
+- **Two-layer capability model** — declarative `ModelCard` (with optional `reasoning`, `modalities`, `tooling`, and `runtime` sections) plus a normalized `ResolvedCapabilityProfile` derived from the card and conservative family defaults. This is the source of truth for prompt rendering, output parsing, tool-call handling, and the `/v1/models` `resolved_capabilities` field.
+- **Phase 2 thinking contract** — `enable_thinking`, `reasoning_effort`, and the dashboard thinking toggle are all driven by `supports_thinking_toggle`, so non-toggleable reasoning models behave correctly without leaking model-specific quirks into client code.
+- **Output parser selection** — model cards declare `output_parser` (`generic`, `gemma4`, `gpt_oss`, `deepseek_v32`, etc.), so reasoning markers are normalized into structured `reasoning_content` per family.
+- **Model store metadata pipeline** — capability resolution feeds `/v1/models` so dashboards and clients can discover thinking, multimodal, and tool support without hardcoding model lists.
+
+### API surface
+
+- **Claude Messages API** — `/v1/messages` adapter, including streaming, tool use, image inputs, and capability-aware thinking controls.
+- **Ollama compatibility** — both `/api/chat` and `/api/generate`, with adapter-side reasoning normalization.
+- **OpenAI Responses API** — `/v1/responses` adapter alongside chat completions.
+- **Embeddings endpoint** for non-chat workloads.
+- **Model store endpoints** — search, add, download, capability resolution, optimization jobs, and registry management, all exposed under stable URLs and documented in the OpenAPI spec.
+- **Cluster-wide config endpoints** — `GET`/`POST` config that gossipsubs to every node and writes back to `skulk.yaml`.
+- **Tracing, downloads, instance previews, and placement endpoints** — distributed-system observability and pre-launch placement inspection that upstream does not expose.
+
+### Dashboard
+
+- **React dashboard (default)** — replaces upstream's Svelte UI with a typed React + styled-components app that ships with the binary. The legacy Svelte dashboard is kept only as a fallback in the repo.
+- **Cluster topology view** with live device icons, GPU stats, network mesh visualization, and connection status banners.
+- **Placement preview / placement manager** for inspecting and choosing valid placements before launching.
+- **Model store browser** with HuggingFace search, family sidebar, model filters, capability badges, recent models, and per-model launch controls.
+- **Reasoning-aware chat UI** that splits inline `<think>` and Gemma `<|channel>` markers into a dedicated thinking panel and merges them with `reasoning_content` deltas from the API.
+- **Image attachments and multimodal chat affordances** for vision models.
+- **Cluster-wide settings panel** that writes to `skulk.yaml` and syncs across nodes via gossipsub.
+- **Light and dark themes** with first-class theme tokens, screenshots in both modes for documentation work.
+
+### Centralized logging and observability
+
+- **Structured JSON stdout** when `logging.enabled` is set, configurable from the dashboard Settings panel and synced cluster-wide.
+- **Vector + VictoriaLogs + Grafana stack** — local Vector log shipper on each node, central VictoriaLogs storage, ready-made Grafana dashboards. Stack definition lives in `deployment/logging/`.
+- **Distributed tracing** opt-in via `EXO_TRACING_ENABLED`.
+
+### Model store
+
+- **Centralized model store host** — one node downloads, the rest of the cluster stages over the LAN.
+- **Persistent registry** with capability resolution and download tracking.
+- **Custom model card support** — add your own model with `POST /models/add`.
+- **Image and embedding model cards** behind feature flags.
+- **Optimization job pipeline** for mlx-optiq mixed-precision weight quantization.
+
+### Cluster operation
+
+- **Cluster-wide settings sync** for KV cache backend, logging, model store host, HF token, and other inference toggles.
+- **Bootstrap peer config from `skulk.yaml`, env, or CLI** for fixed-topology clusters.
+- **Election (bully algorithm) + master/worker split** for indexing events and broadcasting state.
+- **`SKULK_*` environment variables** alongside the legacy `EXO_*` set, so new options can land without colliding with upstream.
+- **`skulk.yaml`** as the canonical config file, with `exo.yaml` kept for backwards compatibility.
+
+### Build, type system, and dev workflow
+
+- **Strict basedpyright** type checking — zero-error policy for new code.
+- **Ruff** linting and **`nix fmt`** formatting in CI.
+- **Nix flake** for reproducible toolchain setup.
+- **Docusaurus docs site** with auto-generated OpenAPI per-endpoint pages and TypeDoc HTML reference for the dashboard, both built from source.
+- **Pre-commit checklist** documented in `CLAUDE.md` and enforced in CI.
+
+### Hardware and platform
+
+- **Apple Silicon as the primary target**, including RDMA over Thunderbolt 5 on supported hardware and matched macOS versions.
+- **Linux supported** (CPU-oriented in this fork; GPU work happens on Apple Silicon).
+
 ## Prerequisites
 
 ### macOS
@@ -106,7 +185,9 @@ Important behavior:
 - **Placement previews**: inspect valid placements before launching a model.
 - **Thinking-aware chat UI**: chat with compatible models and surface reasoning content.
 - **Alternative API compatibility**: OpenAI Chat Completions, OpenAI Responses, Claude Messages, and Ollama.
-- **Experimental inference tuning**: OptiQ and other KV cache backends for long-context and memory experiments.
+- **Rotation-based KV cache backends**: RotorQuant (IsoQuant 3-bit + deferred prefill), OptiQ, TurboQuant, and TurboQuant Adaptive — pick per workload from the dashboard.
+- **Capability-driven thinking contract**: model cards declare reasoning support; the API and dashboard route accordingly.
+- **Experimental inference tuning**: long-context and memory experiments via the KV cache backends above.
 
 ## Dashboard
 
@@ -373,15 +454,18 @@ uv run exo --bootstrap-peers /ip4/192.168.1.20/tcp/5678/p2p/12D3KooW...
 | `EXO_NO_BATCH` | Force sequential generation | `false` |
 | `EXO_OPTIQ_BITS` | Bit width for `optiq` | `4` |
 | `EXO_OPTIQ_FP16_LAYERS` | Edge FP16 layers for `optiq` | `4` |
+| `SKULK_ROTORQUANT_FP16_LAYERS` | Edge FP16 layers for `rotorquant_adaptive` | `4` |
+| `SKULK_ROTORQUANT_DEFER_PREFILL` | Set to `0` to disable deferred prefill (debugging only) | `1` |
 | `EXO_BOOTSTRAP_PEERS` | Comma-separated static peers to dial on startup | None |
 | `HF_TOKEN` | Hugging Face token | None |
 
 Examples:
 
 ```bash
-EXO_OFFLINE=true uv run exo
-EXO_ENABLE_IMAGE_MODELS=true uv run exo
-EXO_KV_CACHE_BACKEND=optiq EXO_OPTIQ_BITS=4 EXO_OPTIQ_FP16_LAYERS=4 uv run exo
+EXO_OFFLINE=true uv run skulk
+EXO_ENABLE_IMAGE_MODELS=true uv run skulk
+EXO_KV_CACHE_BACKEND=optiq EXO_OPTIQ_BITS=4 EXO_OPTIQ_FP16_LAYERS=4 uv run skulk
+SKULK_KV_CACHE_BACKEND=rotorquant_adaptive SKULK_ROTORQUANT_FP16_LAYERS=4 uv run skulk
 ```
 
 ## RDMA on macOS

dashboard-react/src/components/layout/SettingsPanel.tsx

@@ -449,16 +449,20 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
                 filled
                 content={
                   `• Default — No cache quantization. Best baseline quality, highest memory use.\n` +
-                  `• OptiQ — Rotation-based quantization via mlx-optiq. Best long-context quality.\n` +
+                  `• RotorQuant Adaptive — IsoQuant 3-bit with deferred prefill, FP16 edge layers. Recommended.\n` +
+                  `• RotorQuant — IsoQuant 3-bit on all KV layers. Most aggressive compression with deferred prefill.\n` +
+                  `• OptiQ — Rotation-based quantization via mlx-optiq. Good long-context quality, no GQA support.\n` +
                   `• TurboQuant Adaptive — Quantizes middle KV layers, keeps edge layers in FP16. Proven stable.\n` +
-                  `• TurboQuant — Quantizes all KV layers. Most aggressive compression, higher quality risk.\n` +
+                  `• TurboQuant — Quantizes all KV layers. Most aggressive non-rotorquant compression.\n` +
                   `• MLX Quantized — MLX's built-in cache quantization.\n\n` +
-                  `Takes effect on next model launch. Incompatible models fall back to Default automatically.`
+                  `Takes effect on next model launch. OptiQ falls back to Default for unsupported architectures; other backends will error on incompatible models.`
                 }
               />
             </FieldLabel>
             <Select value={kvBackend} onChange={(e) => setKvBackend(e.target.value)} disabled={!!envOverride}>
               <option value="default">Default (no quantization)</option>
+              <option value="rotorquant_adaptive">RotorQuant Adaptive (recommended)</option>
+              <option value="rotorquant">RotorQuant</option>
               <option value="optiq">OptiQ (rotation-based)</option>
               <option value="turboquant_adaptive">TurboQuant Adaptive</option>
               <option value="turboquant">TurboQuant</option>
@@ -467,7 +471,7 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
             {envOverride ? (
               <HintText>Overridden by SKULK_KV_CACHE_BACKEND environment variable. Remove the env var to configure here.</HintText>
             ) : (
-              <HintText>Changes take effect on the next model launch. Models with incompatible architectures (GQA, non-power-of-two head_dim) will automatically fall back to default.</HintText>
+              <HintText>Changes take effect on the next model launch. OptiQ falls back to Default for unsupported architectures; other backends will error on incompatible models.</HintText>
             )}
           </Fieldset>
 

src/exo/store/config.py

@@ -235,7 +235,13 @@ class InferenceConfig(FrozenModel):
     """
 
     kv_cache_backend: Literal[
-        "default", "mlx_quantized", "turboquant", "turboquant_adaptive", "optiq"
+        "default",
+        "mlx_quantized",
+        "turboquant",
+        "turboquant_adaptive",
+        "optiq",
+        "rotorquant",
+        "rotorquant_adaptive",
     ] = "default"
 
 

src/exo/worker/engines/mlx/cache.py

@@ -23,12 +23,19 @@
     KV_CACHE_BITS,
     OPTIQ_BITS,
     OPTIQ_FP16_LAYERS,
+    ROTORQUANT_DEFER_PREFILL,
+    ROTORQUANT_FP16_LAYERS,
     TURBOQUANT_FP16_LAYERS,
     TURBOQUANT_K_BITS,
     TURBOQUANT_V_BITS,
     VALID_KV_CACHE_BACKENDS,
     KVCacheBackend,
 )
+from exo.worker.engines.mlx.rotorquant import (
+    make_rotorquant_adaptive_cache,
+    make_rotorquant_cache_from_template,
+)
+from exo.worker.engines.mlx.rotorquant.tables import ISO3_BLOCK_SIZE
 from exo.worker.engines.mlx.turboquant import (
     make_turboquant_adaptive_cache,
     make_turboquant_cache_from_template,
@@ -417,6 +424,13 @@ def get_memory_used_percentage() -> float:
     return float(mem.percent / 100)
 
 
+def _make_default_cache(model: Model) -> KVCacheType:
+    """Build the default (unquantized) KV cache for a model."""
+    if hasattr(model, "make_cache"):
+        return model.make_cache()  # type: ignore
+    return [KVCache() for _ in model.layers]
+
+
 def make_kv_cache(
     model: Model, max_kv_size: int | None = None, keep: int = 0
 ) -> KVCacheType:
@@ -579,12 +593,46 @@ def _is_power_of_two(n: int) -> bool:
             for i, _ in enumerate(model.layers)
         ]
 
-    if hasattr(model, "make_cache"):
-        logger.info("Using MLX LM's make cache")
-        return model.make_cache()  # type: ignore
+    if backend in ("rotorquant", "rotorquant_adaptive"):
+        # Preflight: RotorQuant requires head_dim divisible by the
+        # IsoQuant block size (128). Models with smaller heads (e.g. 64-d)
+        # would crash at first token; fall back to default instead.
+        if len(model.layers) > 0:
+            first_layer = model.layers[0]
+            attn = getattr(first_layer, "self_attn", first_layer)
+            model_head_dim: int = getattr(attn, "head_dim", ISO3_BLOCK_SIZE)
+        else:
+            model_head_dim = ISO3_BLOCK_SIZE
+        if model_head_dim % ISO3_BLOCK_SIZE != 0:
+            logger.warning(
+                f"RotorQuant requires head_dim divisible by {ISO3_BLOCK_SIZE}, "
+                f"but this model has head_dim={model_head_dim}; "
+                f"falling back to default KV cache"
+            )
+            return _make_default_cache(model)
+
+        if backend == "rotorquant":
+            logger.info(
+                f"Using rotorquant KV cache (defer_prefill={ROTORQUANT_DEFER_PREFILL})"
+            )
+            return make_rotorquant_cache_from_template(
+                model,
+                defer_prefill=ROTORQUANT_DEFER_PREFILL,
+            )
+
+        logger.info(
+            f"Using rotorquant adaptive KV cache "
+            f"(fp16_layers={ROTORQUANT_FP16_LAYERS}, "
+            f"defer_prefill={ROTORQUANT_DEFER_PREFILL})"
+        )
+        return make_rotorquant_adaptive_cache(
+            model,
+            fp16_layers=ROTORQUANT_FP16_LAYERS,
+            defer_prefill=ROTORQUANT_DEFER_PREFILL,
+        )
 
     logger.info("Using default KV cache")
-    return [KVCache() for _ in model.layers]
+    return _make_default_cache(model)
 
 
 def get_kv_cache_backend() -> KVCacheBackend:

src/exo/worker/engines/mlx/constants.py

@@ -23,6 +23,8 @@
     "turboquant",
     "turboquant_adaptive",
     "optiq",
+    "rotorquant",
+    "rotorquant_adaptive",
 ]
 DEFAULT_KV_CACHE_BACKEND: KVCacheBackend = "default"
 VALID_KV_CACHE_BACKENDS: tuple[KVCacheBackend, ...] = (
@@ -31,6 +33,8 @@
     "turboquant",
     "turboquant_adaptive",
     "optiq",
+    "rotorquant",
+    "rotorquant_adaptive",
 )
 _kv_cache_backend_value = preferred_env_value(
     "SKULK_KV_CACHE_BACKEND",
@@ -56,6 +60,20 @@
 DEFAULT_TURBOQUANT_V_BITS: int = 4
 OPTIQ_BITS: int = int(os.environ.get("EXO_OPTIQ_BITS", "4"))
 OPTIQ_FP16_LAYERS: int = int(os.environ.get("EXO_OPTIQ_FP16_LAYERS", "4"))
+ROTORQUANT_FP16_LAYERS: int = int(
+    os.environ.get("SKULK_ROTORQUANT_FP16_LAYERS", os.environ.get("EXO_ROTORQUANT_FP16_LAYERS", "4"))
+)
+# Deferred prefill keeps K/V in fp16 during prompt processing and flushes
+# the buffer to compressed storage on the first decode token. It is the
+# load-bearing accuracy improvement of this backend; only disable for
+# debugging.
+ROTORQUANT_DEFER_PREFILL: bool = (
+    os.environ.get(
+        "SKULK_ROTORQUANT_DEFER_PREFILL",
+        os.environ.get("EXO_ROTORQUANT_DEFER_PREFILL", "1"),
+    )
+    not in ("0", "false", "False", "")
+)
 
 DEFAULT_TOP_LOGPROBS: int = 5
 

src/exo/worker/engines/mlx/rotorquant/__init__.py

@@ -0,0 +1,30 @@
+"""RotorQuant KV cache backend (IsoQuant variant).
+
+Pure-MLX port of the IsoQuant 3-bit KV cache compression from
+johndpope/llama-cpp-turboquant (MIT) and scrya-com/rotorquant (MIT).
+
+Key properties vs the older TurboQuant native backend:
+- Block-diagonal 4D quaternion rotations instead of randomized Hadamard
+- Norm-correction trick for unbiased magnitudes after centroid quantization
+- Optional deferred prefill: K/V stay in fp16 during prompt processing,
+  flushed to compressed storage on the first decode-shaped call
+- GQA-native (compression is per-(kv_head, token), Q heads fan out at SDPA)
+
+The backend stores indices and norms; ``update_and_fetch`` returns fully
+dequantized fp16 K/V to standard ``mx.fast.scaled_dot_product_attention``.
+The centroid-space attention optimization from OptiQ is intentionally
+deferred to a follow-up; v1 prioritizes correctness and the deferred-prefill
+quality win over the rotated-space SDPA perf win.
+"""
+
+from exo.worker.engines.mlx.rotorquant.cache import (
+    RotorQuantKVCache,
+    make_rotorquant_adaptive_cache,
+    make_rotorquant_cache_from_template,
+)
+
+__all__ = [
+    "RotorQuantKVCache",
+    "make_rotorquant_adaptive_cache",
+    "make_rotorquant_cache_from_template",
+]