Add RotorQuant KV cache backend with deferred prefill on Metal

README.md

@@ -20,11 +20,90 @@ a more modern dashboard, richer API workflows, sophisticated cache quantization,
 - Form a small cluster of Macs and split larger models across them.
 - Use a central model store so the cluster downloads once and stages locally.
 - Talk to the cluster through OpenAI Chat Completions, OpenAI Responses, Claude Messages, or Ollama-compatible APIs.
+- Push KV cache memory down hard with rotation-based 3-bit quantization (RotorQuant, OptiQ, TurboQuant) and pick a backend per workload from the dashboard.
+- Use a model-aware reasoning contract that handles toggleable and non-toggleable thinking models without baking assumptions into client code.
 - Experiment with advanced placement modes, RDMA, and KV cache backends when you are ready.
 - Run non-chat workloads such as embeddings and other specialized model flows.
 - Build TTS-oriented and other API-driven workflows on top of the cluster.
 - Actually use your cluster for real inference workloads instead of treating it as a demo.
 
+## Everything Different About Skulk
+
+This is the running list of where Skulk diverges from upstream [exo](https://github.com/exo-explore/exo). It is a living section — every meaningful change should land here when it ships, so anyone evaluating Skulk can see the surface area at a glance.
+
+### Inference and KV cache
+
+- **RotorQuant KV cache backend** — pure-MLX port of IsoQuant 3-bit (block-diagonal quaternion rotations + Lloyd-Max centroids) with **deferred prefill on Metal**, a contribution that does not exist in any upstream project (the llama.cpp fork ships it CUDA-only). GQA-native; no fallback for grouped-query models. See [docs/kv-cache-backends.md](docs/kv-cache-backends.md).
+- **TurboQuant native and adaptive backends** — randomized Hadamard rotation + Lloyd-Max centroids, with an adaptive variant that keeps edge attention layers in fp16 for accuracy.
+- **OptiQ KV cache integration** — wraps `mlx-optiq`'s rotated-space attention path so the rotation cost stays out of the per-token loop on supported (non-GQA) models.
+- **OptiQ mixed-precision weight quantization pipeline** — async wrapper around `mlx-optiq`'s sensitivity analysis and KL-divergence per-layer bit allocation, exposed as a model-store optimization job.
+- **KV prefix cache with snapshot/restore** — LRU-evicted prompt-prefix cache that snapshots SSM and rotating-window cache states so prefix matches are reusable across conversation turns even for hybrid Mamba/Transformer architectures.
+- **Pipeline-parallel prefill for short prompts** — pipelined models now route every prefill through the pipeline path, fixing prior warmup hangs on Gemma-class models.
+- **Force-sequential fallback** — quantized backends transparently fall back to a sequential generator when batch/history mode is incompatible with their cache layout.
+
+### Model capability system
+
+- **Two-layer capability model** — declarative `ModelCard` (with optional `reasoning`, `modalities`, `tooling`, and `runtime` sections) plus a normalized `ResolvedCapabilityProfile` derived from the card and conservative family defaults. This is the source of truth for prompt rendering, output parsing, tool-call handling, and the `/v1/models` `resolved_capabilities` field.
+- **Phase 2 thinking contract** — `enable_thinking`, `reasoning_effort`, and the dashboard thinking toggle are all driven by `supports_thinking_toggle`, so non-toggleable reasoning models behave correctly without leaking model-specific quirks into client code.
+- **Output parser selection** — model cards declare `output_parser` (`generic`, `gemma4`, `gpt_oss`, `deepseek_v32`, etc.), so reasoning markers are normalized into structured `reasoning_content` per family.
+- **Model store metadata pipeline** — capability resolution feeds `/v1/models` so dashboards and clients can discover thinking, multimodal, and tool support without hardcoding model lists.
+
+### API surface
+
+- **Claude Messages API** — `/v1/messages` adapter, including streaming, tool use, image inputs, and capability-aware thinking controls.
+- **Ollama compatibility** — both `/api/chat` and `/api/generate`, with adapter-side reasoning normalization.
+- **OpenAI Responses API** — `/v1/responses` adapter alongside chat completions.
+- **Embeddings endpoint** for non-chat workloads.
+- **Model store endpoints** — search, add, download, capability resolution, optimization jobs, and registry management, all exposed under stable URLs and documented in the OpenAPI spec.
+- **Cluster-wide config endpoints** — `GET`/`POST` config that gossipsubs to every node and writes back to `skulk.yaml`.
+- **Tracing, downloads, instance previews, and placement endpoints** — distributed-system observability and pre-launch placement inspection that upstream does not expose.
+
+### Dashboard
+
+- **React dashboard (default)** — replaces upstream's Svelte UI with a typed React + styled-components app that ships with the binary. The legacy Svelte dashboard is kept only as a fallback in the repo.
+- **Cluster topology view** with live device icons, GPU stats, network mesh visualization, and connection status banners.
+- **Placement preview / placement manager** for inspecting and choosing valid placements before launching.
+- **Model store browser** with HuggingFace search, family sidebar, model filters, capability badges, recent models, and per-model launch controls.
+- **Reasoning-aware chat UI** that splits inline `<think>` and Gemma `<|channel>` markers into a dedicated thinking panel and merges them with `reasoning_content` deltas from the API.
+- **Image attachments and multimodal chat affordances** for vision models.
+- **Cluster-wide settings panel** that writes to `skulk.yaml` and syncs across nodes via gossipsub.
+- **Light and dark themes** with first-class theme tokens, screenshots in both modes for documentation work.
+
+### Centralized logging and observability
+
+- **Structured JSON stdout** when `logging.enabled` is set, configurable from the dashboard Settings panel and synced cluster-wide.
+- **Vector + VictoriaLogs + Grafana stack** — local Vector log shipper on each node, central VictoriaLogs storage, ready-made Grafana dashboards. Stack definition lives in `deployment/logging/`.
+- **Distributed tracing** opt-in via `EXO_TRACING_ENABLED`.
+
+### Model store
+
+- **Centralized model store host** — one node downloads, the rest of the cluster stages over the LAN.
+- **Persistent registry** with capability resolution and download tracking.
+- **Custom model card support** — add your own model with `POST /models/add`.
+- **Image and embedding model cards** behind feature flags.
+- **Optimization job pipeline** for mlx-optiq mixed-precision weight quantization.
+
+### Cluster operation
+
+- **Cluster-wide settings sync** for KV cache backend, logging, model store host, HF token, and other inference toggles.
+- **Bootstrap peer config from `skulk.yaml`, env, or CLI** for fixed-topology clusters.
+- **Election (bully algorithm) + master/worker split** for indexing events and broadcasting state.
+- **`SKULK_*` environment variables** alongside the legacy `EXO_*` set, so new options can land without colliding with upstream.
+- **`skulk.yaml`** as the canonical config file, with `exo.yaml` kept for backwards compatibility.
+
+### Build, type system, and dev workflow
+
+- **Strict basedpyright** type checking — zero-error policy for new code.
+- **Ruff** linting and **`nix fmt`** formatting in CI.
+- **Nix flake** for reproducible toolchain setup.
+- **Docusaurus docs site** with auto-generated OpenAPI per-endpoint pages and TypeDoc HTML reference for the dashboard, both built from source.
+- **Pre-commit checklist** documented in `CLAUDE.md` and enforced in CI.
+
+### Hardware and platform
+
+- **Apple Silicon as the primary target**, including RDMA over Thunderbolt 5 on supported hardware and matched macOS versions.
+- **Linux supported** (CPU-oriented in this fork; GPU work happens on Apple Silicon).
+
 ## Prerequisites
 
 ### macOS
@@ -106,7 +185,9 @@ Important behavior:
 - **Placement previews**: inspect valid placements before launching a model.
 - **Thinking-aware chat UI**: chat with compatible models and surface reasoning content.
 - **Alternative API compatibility**: OpenAI Chat Completions, OpenAI Responses, Claude Messages, and Ollama.
-- **Experimental inference tuning**: OptiQ and other KV cache backends for long-context and memory experiments.
+- **Rotation-based KV cache backends**: RotorQuant (IsoQuant 3-bit + deferred prefill), OptiQ, TurboQuant, and TurboQuant Adaptive — pick per workload from the dashboard.
+- **Capability-driven thinking contract**: model cards declare reasoning support; the API and dashboard route accordingly.
+- **Experimental inference tuning**: long-context and memory experiments via the KV cache backends above.
 
 ## Dashboard
 
@@ -373,15 +454,18 @@ uv run exo --bootstrap-peers /ip4/192.168.1.20/tcp/5678/p2p/12D3KooW...
 | `EXO_NO_BATCH` | Force sequential generation | `false` |
 | `EXO_OPTIQ_BITS` | Bit width for `optiq` | `4` |
 | `EXO_OPTIQ_FP16_LAYERS` | Edge FP16 layers for `optiq` | `4` |
+| `SKULK_ROTORQUANT_FP16_LAYERS` | Edge FP16 layers for `rotorquant_adaptive` | `4` |
+| `SKULK_ROTORQUANT_DEFER_PREFILL` | Set to `0` to disable deferred prefill (debugging only) | `1` |
 | `EXO_BOOTSTRAP_PEERS` | Comma-separated static peers to dial on startup | None |
 | `HF_TOKEN` | Hugging Face token | None |
 
 Examples:
 
 ```bash
-EXO_OFFLINE=true uv run exo
-EXO_ENABLE_IMAGE_MODELS=true uv run exo
-EXO_KV_CACHE_BACKEND=optiq EXO_OPTIQ_BITS=4 EXO_OPTIQ_FP16_LAYERS=4 uv run exo
+EXO_OFFLINE=true uv run skulk
+EXO_ENABLE_IMAGE_MODELS=true uv run skulk
+EXO_KV_CACHE_BACKEND=optiq EXO_OPTIQ_BITS=4 EXO_OPTIQ_FP16_LAYERS=4 uv run skulk
+SKULK_KV_CACHE_BACKEND=rotorquant_adaptive SKULK_ROTORQUANT_FP16_LAYERS=4 uv run skulk
 ```
 
 ## RDMA on macOS

dashboard-react/src/components/layout/SettingsPanel.tsx

@@ -449,16 +449,20 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
                 filled
                 content={
                   `• Default — No cache quantization. Best baseline quality, highest memory use.\n` +
-                  `• OptiQ — Rotation-based quantization via mlx-optiq. Best long-context quality.\n` +
+                  `• RotorQuant Adaptive — IsoQuant 3-bit with deferred prefill, FP16 edge layers. Recommended.\n` +
+                  `• RotorQuant — IsoQuant 3-bit on all KV layers. Most aggressive compression with deferred prefill.\n` +
+                  `• OptiQ — Rotation-based quantization via mlx-optiq. Good long-context quality, no GQA support.\n` +
                   `• TurboQuant Adaptive — Quantizes middle KV layers, keeps edge layers in FP16. Proven stable.\n` +
-                  `• TurboQuant — Quantizes all KV layers. Most aggressive compression, higher quality risk.\n` +
+                  `• TurboQuant — Quantizes all KV layers. Most aggressive non-rotorquant compression.\n` +
                   `• MLX Quantized — MLX's built-in cache quantization.\n\n` +
-                  `Takes effect on next model launch. Incompatible models fall back to Default automatically.`
+                  `Takes effect on next model launch. OptiQ falls back to Default for unsupported architectures; other backends will error on incompatible models.`
                 }
               />
             </FieldLabel>
             <Select value={kvBackend} onChange={(e) => setKvBackend(e.target.value)} disabled={!!envOverride}>
               <option value="default">Default (no quantization)</option>
+              <option value="rotorquant_adaptive">RotorQuant Adaptive (recommended)</option>
+              <option value="rotorquant">RotorQuant</option>
               <option value="optiq">OptiQ (rotation-based)</option>
               <option value="turboquant_adaptive">TurboQuant Adaptive</option>
               <option value="turboquant">TurboQuant</option>
@@ -467,7 +471,7 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
             {envOverride ? (
               <HintText>Overridden by SKULK_KV_CACHE_BACKEND environment variable. Remove the env var to configure here.</HintText>
             ) : (
-              <HintText>Changes take effect on the next model launch. Models with incompatible architectures (GQA, non-power-of-two head_dim) will automatically fall back to default.</HintText>
+              <HintText>Changes take effect on the next model launch. OptiQ falls back to Default for unsupported architectures; other backends will error on incompatible models.</HintText>
             )}
           </Fieldset>
 

src/exo/main.py

@@ -29,6 +29,7 @@
     load_exo_config,
     resolve_config_path,
     resolve_node_staging,
+    update_config_field,
 )
 from exo.store.model_store import ModelStore
 from exo.store.model_store_client import ModelStoreClient, ModelStoreDownloader
@@ -98,12 +99,26 @@ async def create(cls, args: "Args") -> Self:
             "1" if _user_set_kv_backend else ""
         )  # legacy compat
 
-        # Apply inference config to env var so runner subprocesses inherit it.
-        # Env var takes precedence if user set it at launch.
-        if (
+        # Env var is the source of truth for KV backend. When set at launch,
+        # write it back to the config file so they stay in sync. When not
+        # set, apply the config file value to the env var so runner
+        # subprocesses inherit it.
+        if _user_set_kv_backend:
+            launch_backend = os.environ.get(
+                "SKULK_KV_CACHE_BACKEND",
+                os.environ.get("EXO_KV_CACHE_BACKEND", ""),
+            )
+            if launch_backend:
+                if update_config_field("inference", "kv_cache_backend", launch_backend):
+                    logger.info(
+                        f"Synced launch env KV backend to config: kv_cache_backend={launch_backend}"
+                    )
+                # Ensure both env vars are in sync
+                os.environ["SKULK_KV_CACHE_BACKEND"] = launch_backend
+                os.environ["EXO_KV_CACHE_BACKEND"] = launch_backend
+        elif (
             exo_config is not None
             and exo_config.inference is not None
-            and not _user_set_kv_backend
         ):
             os.environ["SKULK_KV_CACHE_BACKEND"] = exo_config.inference.kv_cache_backend
             os.environ["EXO_KV_CACHE_BACKEND"] = (

src/exo/store/config.py

@@ -235,10 +235,38 @@ class InferenceConfig(FrozenModel):
     """
 
     kv_cache_backend: Literal[
-        "default", "mlx_quantized", "turboquant", "turboquant_adaptive", "optiq"
+        "default",
+        "mlx_quantized",
+        "turboquant",
+        "turboquant_adaptive",
+        "optiq",
+        "rotorquant",
+        "rotorquant_adaptive",
     ] = "default"
 
 
+def update_config_field(section: str, key: str, value: object) -> bool:
+    """Update a single field in the config file, preserving all other content.
+
+    Reads the raw YAML, patches ``raw[section][key] = value``, and writes
+    it back.  Returns ``True`` if the file was updated, ``False`` if no
+    config file exists.
+    """
+    path = resolve_config_path()
+    if not path.exists():
+        return False
+    with path.open() as f:
+        raw: dict[str, object] = yaml.safe_load(f) or {}
+    sec = raw.get(section)
+    if not isinstance(sec, dict):
+        sec = {}
+        raw[section] = sec
+    sec[key] = value
+    with path.open("w") as f:
+        yaml.dump(raw, f, default_flow_style=False, sort_keys=False)
+    return True
+
+
 def resolve_config_path() -> Path:
     """Find the config file, preferring ``skulk.yaml`` over legacy ``exo.yaml``."""
     skulk = Path("skulk.yaml")

src/exo/worker/engines/mlx/cache.py

@@ -12,23 +12,30 @@
 )
 from mlx_lm.tokenizer_utils import TokenizerWrapper
 
+from exo.shared.constants import preferred_env_value
 from exo.shared.types.memory import Memory
 from exo.shared.types.mlx import KVCacheType, Model
 from exo.worker.engines.mlx.constants import (
     CACHE_GROUP_SIZE,
     DEFAULT_KV_CACHE_BACKEND,
     DEFAULT_TURBOQUANT_K_BITS,
     DEFAULT_TURBOQUANT_V_BITS,
-    KV_CACHE_BACKEND,
     KV_CACHE_BITS,
     OPTIQ_BITS,
     OPTIQ_FP16_LAYERS,
+    ROTORQUANT_DEFER_PREFILL,
+    ROTORQUANT_FP16_LAYERS,
     TURBOQUANT_FP16_LAYERS,
     TURBOQUANT_K_BITS,
     TURBOQUANT_V_BITS,
     VALID_KV_CACHE_BACKENDS,
     KVCacheBackend,
 )
+from exo.worker.engines.mlx.rotorquant import (
+    make_rotorquant_adaptive_cache,
+    make_rotorquant_cache_from_template,
+)
+from exo.worker.engines.mlx.rotorquant.tables import ISO3_BLOCK_SIZE
 from exo.worker.engines.mlx.turboquant import (
     make_turboquant_adaptive_cache,
     make_turboquant_cache_from_template,
@@ -417,6 +424,13 @@ def get_memory_used_percentage() -> float:
     return float(mem.percent / 100)
 
 
+def _make_default_cache(model: Model) -> KVCacheType:
+    """Build the default (unquantized) KV cache for a model."""
+    if hasattr(model, "make_cache"):
+        return model.make_cache()  # type: ignore
+    return [KVCache() for _ in model.layers]
+
+
 def make_kv_cache(
     model: Model, max_kv_size: int | None = None, keep: int = 0
 ) -> KVCacheType:
@@ -579,19 +593,68 @@ def _is_power_of_two(n: int) -> bool:
             for i, _ in enumerate(model.layers)
         ]
 
-    if hasattr(model, "make_cache"):
-        logger.info("Using MLX LM's make cache")
-        return model.make_cache()  # type: ignore
+    if backend in ("rotorquant", "rotorquant_adaptive"):
+        # Preflight: RotorQuant requires head_dim divisible by the
+        # IsoQuant block size (128). Models with smaller heads (e.g. 64-d)
+        # would crash at first token; fall back to default instead.
+        if len(model.layers) > 0:
+            first_layer = model.layers[0]
+            attn = getattr(first_layer, "self_attn", first_layer)
+            model_head_dim: int = getattr(attn, "head_dim", ISO3_BLOCK_SIZE)
+        else:
+            model_head_dim = ISO3_BLOCK_SIZE
+        if model_head_dim % ISO3_BLOCK_SIZE != 0:
+            logger.warning(
+                f"RotorQuant requires head_dim divisible by {ISO3_BLOCK_SIZE}, "
+                f"but this model has head_dim={model_head_dim}; "
+                f"falling back to default KV cache"
+            )
+            return _make_default_cache(model)
+
+        if backend == "rotorquant":
+            logger.info(
+                f"Using rotorquant KV cache (defer_prefill={ROTORQUANT_DEFER_PREFILL})"
+            )
+            return make_rotorquant_cache_from_template(
+                model,
+                defer_prefill=ROTORQUANT_DEFER_PREFILL,
+            )
+
+        logger.info(
+            f"Using rotorquant adaptive KV cache "
+            f"(fp16_layers={ROTORQUANT_FP16_LAYERS}, "
+            f"defer_prefill={ROTORQUANT_DEFER_PREFILL})"
+        )
+        return make_rotorquant_adaptive_cache(
+            model,
+            fp16_layers=ROTORQUANT_FP16_LAYERS,
+            defer_prefill=ROTORQUANT_DEFER_PREFILL,
+        )
 
     logger.info("Using default KV cache")
-    return [KVCache() for _ in model.layers]
+    return _make_default_cache(model)
 
 
 def get_kv_cache_backend() -> KVCacheBackend:
-    backend = KV_CACHE_BACKEND
+    """Return the active KV cache backend, reading ``os.environ`` at call time.
+
+    The env var is the source of truth — it can be set at launch, updated
+    by config sync via gossipsub, or changed from the dashboard.  Reading
+    it dynamically (instead of the frozen import-time constant) ensures
+    that runtime updates are always visible to the runner.
+    """
+    backend = cast(
+        KVCacheBackend,
+        preferred_env_value(
+            "SKULK_KV_CACHE_BACKEND",
+            "EXO_KV_CACHE_BACKEND",
+            DEFAULT_KV_CACHE_BACKEND,
+        )
+        or DEFAULT_KV_CACHE_BACKEND,
+    )
     if backend not in VALID_KV_CACHE_BACKENDS:
         logger.warning(
-            f"Unknown EXO_KV_CACHE_BACKEND={backend!r}; falling back to {DEFAULT_KV_CACHE_BACKEND!r}"
+            f"Unknown KV_CACHE_BACKEND={backend!r}; falling back to {DEFAULT_KV_CACHE_BACKEND!r}"
         )
         return DEFAULT_KV_CACHE_BACKEND
     return backend