feat(vtuber): add OpenAI-compatible adapter by canyugs · Pull Request #1234 · openabdev/openab

canyugs · 2026-06-28T15:52:03Z

What problem does this solve?

Desktop character apps such as AniCompanion, Open-LLM-VTuber, and ChatVRM already speak OpenAI chat completions, but they usually connect to a raw LLM. This PR lets those skins point at OpenAB instead, so the same UI gets ACP-backed tool use, code editing, memory, MCP, and existing OpenAB steering.

Closes #1233

Discord Discussion URL: https://discord.com/channels/1491295327620169908/1520790210320011274

Architecture

The VTuber adapter now runs inside the unified OpenAB binary:

Skin (AniCompanion / Open-LLM-VTuber / ChatVRM)
  |
  |-- POST /v1/chat/completions  (stream:true, Bearer key)
  |     choices[].delta.content, including inline [emotion] tags
  |
OpenAB unified binary
  |-- crates/openab-gateway/src/adapters/vtuber.rs
  |-- src/unified_adapter.rs
  |-- src/acp -> coding agent (codex / claude / kiro)

No separate gateway process or adapter config block is required for VTuber in unified mode. Set VTUBER_ENABLED=true on the OpenAB process, and the unified HTTP listener exposes the OpenAI-compatible endpoint.

Proposed Solution

OpenAI-Compatible SSE

POST /v1/chat/completions streams OpenAI-compatible chat.completion.chunk events.
Inline [emotion] tags pass through for skins that already parse and strip them before TTS.
A stable vtb_persistent channel reuses one warm ACP session for VTuber traffic, avoiding per-turn cold starts.
Requests are serialized with a stream-owned lock; concurrent requests return 429 with Retry-After: 3.
OpenAI-compatible skins may still send full messages[]; the adapter forwards system/developer instructions plus the latest user turn so assistant history is not duplicated inside the persistent ACP session.

Why this approach?

Zero client changes for skins that already support OpenAI-compatible chat completions.
Unified binary deployment matches the current OpenAB platform-adapter model.
Persistent session reuse preserves the latency win from Add Gemini CLI support with separate Docker image #15 while keeping stream lifetime locking and prompt forwarding explicit.

Validation

cargo test -p openab-gateway vtuber — 7 VTuber tests passed
cargo test -p openab has_unified_platform_env_checks --features unified
cargo check -p openab --features unified
cargo clippy -- -D warnings
cargo test

Notes

docs/vtuber.md documents unified-mode setup only.
docs/config-reference.md lists the VTuber unified environment variables.
Tier-2 WebSocket side-channel work was deferred out of this PR and should land separately when there is a concrete client workflow.

Expose POST /v1/chat/completions (SSE) backed by the OAB agent, so any OpenAI-compatible character skin (AniCompanion, Open-LLM-VTuber, …) gets a real agent with zero client changes. messages[] is flattened into the agent prompt; the agent's streamed reply is re-emitted as OpenAI chat.completion.chunk deltas via a per-request channel.id registry drained by the /ws recv loop. Inline [emotion] tags pass through untouched. Tier 1 of RFC openabdev#1233. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m

smallgun01 · 2026-06-28T16:06:28Z

Additional: Frontend WebSocket Client Demo

Added examples/vtuber-demo/index.html — a minimal WebSocket client reference implementation for VTuber skins.

What it does:

Connects to OAB Gateway via raw WebSocket (ws://host:8080/ws?token=)
Sends openab.gateway.event.v1 (text messages)
Receives openab.gateway.reply.v1 (streaming reply with cursor animation)
Dark theme UI, settings persisted in localStorage

Usage:

Open index.html in browser, configure WS URL and token, connect and chat.

Note:

Currently the gateway's handle_oab_connection does not forward GatewayEvents from WS clients to OAB core — backend update needed for full functionality.

Code:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>OAB VTuber Client</title>
  <style>
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body { font-family: monospace; background: #1a1a2e; color: #e0e0e0; display: flex; flex-direction: column; height: 100vh; }
    #header { padding: 12px 16px; background: #16213e; border-bottom: 1px solid #0f3460; display: flex; align-items: center; gap: 12px; }
    #header h1 { font-size: 16px; color: #e94560; }
    #status { font-size: 12px; padding: 2px 8px; border-radius: 4px; }
    .disconnected { background: #555; }
    .connected { background: #2d6a4f; }
    .error { background: #9b2226; }
    #config { padding: 8px 16px; background: #16213e; border-bottom: 1px solid #0f3460; display: flex; gap: 8px; flex-wrap: wrap; align-items: center; font-size: 12px; }
    #config input { background: #0f3460; border: 1px solid #533483; color: #e0e0e0; padding: 4px 8px; border-radius: 4px; font-family: monospace; font-size: 12px; }
    #config button { background: #533483; border: none; color: #e0e0e0; padding: 4px 12px; border-radius: 4px; cursor: pointer; font-size: 12px; }
    #config button:hover { background: #e94560; }
    #chat { flex: 1; overflow-y: auto; padding: 16px; display: flex; flex-direction: column; gap: 8px; }
    .msg { padding: 8px 12px; border-radius: 6px; max-width: 80%; white-space: pre-wrap; word-break: break-word; font-size: 14px; line-height: 1.5; }
    .msg.user { background: #533483; align-self: flex-end; }
    .msg.assistant { background: #0f3460; align-self: flex-start; }
    .msg.system { background: #2d2d2d; align-self: center; font-size: 11px; color: #888; }
    .msg.error { background: #9b2226; align-self: center; font-size: 12px; }
    #input-area { padding: 12px 16px; background: #16213e; border-top: 1px solid #0f3460; display: flex; gap: 8px; }
    #msg-input { flex: 1; background: #0f3460; border: 1px solid #533483; color: #e0e0e0; padding: 8px 12px; border-radius: 6px; font-family: monospace; font-size: 14px; resize: none; }
    #send-btn { background: #e94560; border: none; color: #fff; padding: 8px 20px; border-radius: 6px; cursor: pointer; font-size: 14px; font-weight: bold; }
    #send-btn:hover { background: #c81d4e; }
    #send-btn:disabled { background: #555; cursor: not-allowed; }
    .cursor { display: inline-block; width: 8px; height: 14px; background: #e94560; animation: blink 0.8s infinite; vertical-align: text-bottom; }
    @keyframes blink { 0%, 100% { opacity: 1; } 50% { opacity: 0; } }
  </style>
</head>
<body>
  <div id="header">
    <h1>OAB VTuber Client</h1>
    <span id="status" class="disconnected">Disconnected</span>
  </div>
  <div id="config">
    <label>WS: <input type="text" id="ws-url" size="40"></label>
    <label>Token: <input type="password" id="ws-token" placeholder="API key" size="20"></label>
    <button id="connect-btn" onclick="toggleConnection()">Connect</button>
  </div>
  <div id="chat"></div>
  <div id="input-area">
    <textarea id="msg-input" rows="2" placeholder="Type a message..." onkeydown="handleKey(event)"></textarea>
    <button id="send-btn" onclick="sendMessage()" disabled>Send</button>
  </div>

  <script>
    const STORAGE_KEY = 'oab-vtuber-config';
    let ws = null;
    let currentAssistantEl = null;
    let cursorEl = null;

    const chatEl = document.getElementById('chat');
    const msgInput = document.getElementById('msg-input');
    const sendBtn = document.getElementById('send-btn');
    const connectBtn = document.getElementById('connect-btn');
    const statusEl = document.getElementById('status');
    const wsUrlInput = document.getElementById('ws-url');
    const wsTokenInput = document.getElementById('ws-token');

    function loadConfig() {
      try {
        const saved = JSON.parse(localStorage.getItem(STORAGE_KEY));
        if (saved) {
          wsUrlInput.value = saved.wsUrl || 'ws://localhost:8080/ws';
          wsTokenInput.value = saved.token || '';
        }
      } catch {
        wsUrlInput.value = 'ws://localhost:8080/ws';
      }
    }

    function saveConfig() {
      localStorage.setItem(STORAGE_KEY, JSON.stringify({
        wsUrl: wsUrlInput.value.trim(),
        token: wsTokenInput.value.trim()
      }));
    }

    loadConfig();

    function addMessage(text, role) {
      const el = document.createElement('div');
      el.className = `msg ${role}`;
      el.textContent = text;
      chatEl.appendChild(el);
      chatEl.scrollTop = chatEl.scrollHeight;
      return el;
    }

    function showCursor(parentEl) {
      removeCursor();
      cursorEl = document.createElement('span');
      cursorEl.className = 'cursor';
      parentEl.appendChild(cursorEl);
    }

    function removeCursor() {
      if (cursorEl) {
        cursorEl.remove();
        cursorEl = null;
      }
    }

    function setStatus(state, text) {
      statusEl.className = state;
      statusEl.textContent = text;
    }

    function toggleConnection() {
      if (ws && ws.readyState === WebSocket.OPEN) {
        ws.close();
      } else {
        connectWS();
      }
    }

    function connectWS() {
      const baseUrl = wsUrlInput.value.trim();
      const token = wsTokenInput.value.trim();
      if (!baseUrl) return addMessage('Please enter WebSocket URL', 'error');

      saveConfig();
      const url = token ? `${baseUrl}?token=${encodeURIComponent(token)}` : baseUrl;
      addMessage(`Connecting to ${url}...`, 'system');

      try {
        ws = new WebSocket(url);
      } catch (e) {
        addMessage(`Connection error: ${e.message}`, 'error');
        return;
      }

      ws.onopen = () => {
        setStatus('connected', 'Connected');
        connectBtn.textContent = 'Disconnect';
        sendBtn.disabled = false;
        addMessage('Connected to OAB Gateway', 'system');
      };

      ws.onmessage = (event) => {
        try {
          const reply = JSON.parse(event.data);

          if (reply.schema === 'openab.gateway.reply.v1') {
            const content = reply.content;
            if (content && content.type === 'text' && content.text) {
              if (!currentAssistantEl) {
                currentAssistantEl = addMessage('', 'assistant');
              }
              currentAssistantEl.textContent += content.text;
              showCursor(currentAssistantEl);
              chatEl.scrollTop = chatEl.scrollHeight;
            }

            if (reply.done) {
              removeCursor();
              currentAssistantEl = null;
            }
          } else if (reply.type === 'done' || reply.done === true) {
            removeCursor();
            currentAssistantEl = null;
          } else {
            addMessage(`[Unknown reply] ${event.data}`, 'system');
          }
        } catch {
          if (!currentAssistantEl) {
            currentAssistantEl = addMessage('', 'assistant');
          }
          currentAssistantEl.textContent += event.data;
          showCursor(currentAssistantEl);
          chatEl.scrollTop = chatEl.scrollHeight;
        }
      };

      ws.onclose = () => {
        setStatus('disconnected', 'Disconnected');
        connectBtn.textContent = 'Connect';
        sendBtn.disabled = true;
        removeCursor();
        currentAssistantEl = null;
        addMessage('Disconnected', 'system');
      };

      ws.onerror = () => {
        setStatus('error', 'Error');
        addMessage('WebSocket error', 'error');
      };
    }

    function sendMessage() {
      const text = msgInput.value.trim();
      if (!text || !ws || ws.readyState !== WebSocket.OPEN) return;

      addMessage(text, 'user');
      currentAssistantEl = null;
      removeCursor();

      const payload = {
        schema: 'openab.gateway.event.v1',
        platform: 'vtuber',
        content: {
          type: 'text',
          text: text
        }
      };

      ws.send(JSON.stringify(payload));
      msgInput.value = '';
    }

    function handleKey(e) {
      if (e.key === 'Enter' && !e.shiftKey) {
        e.preventDefault();
        sendMessage();
      }
    }
  </script>
</body>
</html>

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

canyugs · 2026-06-28T17:49:13Z

Tier-1 complete ✅

All validation items checked:

Unit tests (config, flatten_messages, delta_suffix incl. multibyte) — 257 passed
Full e2e against real gateway binary (fake agent)
Real ACP agent e2e (kiro-cli driving live LLM)
Cloud deployment e2e (Zeabur, Tencent Tokyo) — curl -sN with Bearer auth → streamed OpenAI deltas → [DONE]
AniCompanion e2e (macOS VRM app) — zero code changes, pointed at cloud gateway, sent messages and received replies from claude-agent-acp

CI: all checks and smoke tests passing.

Next: Tier-2

Tier-2 RFC opened as #1235 — WebSocket side-channel (/v1/vtuber/ws) for agent-state push, tool visibility, emotion, and ambient notifications. Design is informed by prior art from Open-LLM-VTuber, clawd-on-desk, and VTube Studio API.

Tier-2 adds GET /v1/vtuber/ws — a persistent WebSocket that pushes agent_state, emotion, and notification events derived from GatewayReply commands. VTuber skins connect once and receive real-time state updates (thinking/working/idle, tool usage, emotion tags) without polling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m

canyugs · 2026-06-28T20:02:52Z

Tier-2 WebSocket event stream pushed (acd3e38). Pending: e2e testing with a live VTuber skin.

Replace Vec<WsClient> with HashMap<u64, WsClient> and an AtomicU64 counter. The old Vec+index scheme broke when broadcast() called swap_remove on dead clients — surviving clients' stored indices became stale, routing subscribe/pong to the wrong connection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m

F2: Cap in-flight /v1/chat/completions at 32 by checking vtuber_pending size before accepting — returns 429 when full. F3: Emit SSE comment `: waiting for agent` after 10s of silence, giving clients an early signal before the 180s hard timeout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m

canyugs · 2026-06-29T02:18:26Z

Review findings addressed:

F1 (client index bug): Fixed in c154dc1 — Vec<WsClient> + index → HashMap<u64, WsClient> + AtomicU64. No more stale index after swap_remove.
F2 (rate limit): Fixed in 50c252e — cap at 32 in-flight requests via vtuber_pending.len() check, returns 429.
F3 (idle warning): Fixed in 50c252e — SSE comment : waiting for agent emitted after 10s of silence, before the 180s hard timeout.
F1 (duplicate AppState): Deferred — this pattern is shared across all adapters; refactoring serve() to delegate to AppState::from_env() should be a separate PR.
F3 (emotion intensity): Deferred — hardcoded 1.0 is sufficient until a skin needs variable intensity ([tag:0.8] format).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m

canyugs · 2026-06-29T04:11:15Z

All findings resolved, clippy CI fixed in c1b9495. Ready for merge — pending e2e testing with a live VTuber skin.

Rebased persistent channel session reuse on Can's cleaned-up feat/vtuber-adapter (Tier-1 HTTP SSE only, no WS side-channel). Changes retained from original PR: - Fixed channel_id "vtb_persistent" for OAB session pool reuse - try_lock + 429 + Retry-After:3 for serialised access - Persistent channel kept in registry after reply - Improved timeout warning message Dropped from original (Can's base already removed these): - vtuber_ws_clients (Tier-2 WS, deferred upstream) - vtuber_persistent_channel field (dead code, already removed) Co-authored-by: smallgun01 <smallgun01@users.noreply.github.com>

canyugs · 2026-07-03T05:49:39Z

Latest head looks good from my pass.

I rechecked the final diff after Tier-2 was deferred: this is now scoped to the OpenAI-compatible SSE adapter, persistent VTuber session reuse, serialized turns with 429/Retry-After, constant-time bearer auth comparison, and the docs/config reference updates. No blocking issues found.

CI and Docker smoke are green now: 34 pass, 2 skipped, 0 pending/failing. Ball back to reviewer/maintainer for final approval/merge.

chaodu-agent · 2026-07-03T07:35:32Z

LGTM ✅ — Well-structured OpenAI-compatible adapter that follows existing conventions.

What This PR Does

Adds a VTuber platform adapter so character skins (AniCompanion, Open-LLM-VTuber, ChatVRM) that speak OpenAI chat completions can connect directly to OpenAB and get full agent capabilities (tool use, MCP, memory) without code changes on the skin side.

How It Works

The adapter exposes POST /v1/chat/completions (SSE) in the unified binary. Incoming messages are forwarded to a persistent warm ACP session (vtb_persistent channel) to avoid cold starts. Replies stream back as chat.completion.chunk deltas via snapshot-diffing. Requests are serialized with a mutex; concurrent requests get 429 Retry-After: 3. Auth uses constant-time Bearer key comparison.

Findings

#	Severity	Finding	Location
1	🟢	Constant-time auth via `subtle::ConstantTimeEq` — proper security practice	`vtuber.rs:165`
2	🟢	Feature-flag gated with `#[cfg(feature = "vtuber")]` throughout — zero cost when disabled	Cargo.toml, lib.rs, main.rs
3	🟢	Comprehensive docs with env var table, troubleshooting, and curl example	`docs/vtuber.md`
4	🟢	7 tests covering auth, streaming enforcement, serialization, idle timeout, and message formatting	`vtuber.rs` tests
5	🟢	Clean integration into both `serve()` and unified-binary paths with proper reply routing	`lib.rs`, `main.rs`, `unified_adapter.rs`

Finding Details

🟢 F1: Constant-time auth comparison

Uses subtle::ConstantTimeEq for Bearer key validation — prevents timing side-channels on the auth key.

🟢 F2: Feature-flag discipline

Every vtuber integration point is gated behind #[cfg(feature = "vtuber")]. When the feature is not compiled in, there is zero binary size or runtime cost. The flag is included in the unified feature set and added to both root and gateway Cargo.toml.

🟢 F3: Complete documentation

docs/vtuber.md covers prerequisites, setup steps, Docker/K8s examples, skin configuration, environment variable reference, emotion tag pass-through semantics, limitations, and troubleshooting — all in 142 lines.

🟢 F4: Good test coverage

Tests validate: auth key constant-time comparison, non-streaming rejection, in-flight cap enforcement, request serialization for stream lifetime, message flattening, persistent session prompt extraction, and reply stream idle close.

🟢 F5: Dual-path integration

Reply routing is wired in both the WebSocket handler (lib.rs handle_oab_connection) and the unified adapter dispatch (unified_adapter.rs), matching the pattern used by all other platform adapters.

Baseline Check

PR opened: 2026-07-03
Main already has: feishu, googlechat, line, teams, telegram, wecom adapters; unified binary framework; src/unified_adapter.rs
Net-new value: Entirely new VTuber adapter — no vtuber/vtb code exists on main. Adds OpenAI-compatible SSE endpoint, persistent session reuse, tool-status suppression, and full docs.

What's Good (🟢)

Follows the existing adapter pattern precisely (feature flag, AppState fields, route registration, reply dispatch)
Persistent session reuse avoids per-turn agent cold starts
Tool-status suppression (parse_pure_tool_status) keeps skin output clean
Rate limiting via both mutex (one-at-a-time) and in-flight cap (32 max pending)
Warning when VTUBER_AUTH_KEY is unset rather than silently running unauthenticated
VTUBER_PATH and VTUBER_REPLY_TAIL_IDLE_MS env vars provide operational tunability
Rustfmt reformatting in adjacent files is cosmetic-only and improves readability

canyugs marked this pull request as ready for review June 28, 2026 17:33

canyugs requested a review from thepagent as a code owner June 28, 2026 17:33

Copilot AI review requested due to automatic review settings June 28, 2026 17:33

Copilot AI reviewed Jun 28, 2026

canyugs mentioned this pull request Jun 28, 2026

RFC: VTuber adapter Tier-2 — WebSocket agent-state push #1235

Open

canyugs requested a review from Copilot June 28, 2026 17:48

Copilot AI reviewed Jun 28, 2026

github-actions Bot added the pending-contributor label Jun 28, 2026