diff --git a/README.md b/README.md index c2c4722..9768917 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,9 @@ What is actually in the tree: - Specs for the voice/desktop goal, Spark memory budget, and follow-on work (`specs/`) -> **Used by:** the companion [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent) repo — a local voice front-end (STT/TTS on a DGX Spark) that uses this package as its agent brain. +> **Used by:** [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent) +> executes [spec 008](specs/008-local-tts-lenovo-go-spike/spec.md) (the Lenovo Go +> local-tts spike). This package is `get_agent()` only. > > **Fleet ops:** the workstation's multi-profile fleet (orchestrator, architect, > researcher, coder, reviewer) is documented in local Hermes docs at @@ -160,6 +162,7 @@ examples/ Non-interactive snippets - **[Architecture](docs/architecture.md)** — layering of this package - **[Development](docs/development.md)** — venv, chat, common commands - **[Specs index](specs/README.md)** — design and planning already in this repo +- **[Spec 008](specs/008-local-tts-lenovo-go-spike/spec.md)** — local-tts spike; implemented in [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent) - Workstation fleet operating manual (Hermes, not vendored): `~/.hermes/docs/agentic-workflow.md` diff --git a/specs/001-voice-dgx-spark-agent/plan.md b/specs/001-voice-dgx-spark-agent/plan.md index 21f5141..ae7eb19 100644 --- a/specs/001-voice-dgx-spark-agent/plan.md +++ b/specs/001-voice-dgx-spark-agent/plan.md @@ -41,7 +41,7 @@ We will build a **layered voice agent** with clear separation of concerns: **Key Principle**: The voice layer is "dumb but fast". The LangGraph agent is the intelligent brain that decides *what* to say and *when* to use memory/tools. -**Parallel lightweight spike (2026-06-12)**: A low-commitment hardware prototype loop lives in `conversational-voice-agent`: named-pipe (or Teams button) trigger, direct ALSA arecord/aplay on the specific Lenovo Go devices, `riva.client` streaming ASR with real partials, raw HID for the Teams LED, and a `speak()` / speak-pipe entry point for replies. It is used for rapid prototyping of the natural voice "feel" on real desk hardware and for exercising the *existing* agent + Supermemory (via the minimal seams) while the primary `thelab_langchain.voice` (orchestrator + audio + VAD) and telephony production path mature. See the local-tts handoff and its spike spec. The spike deliberately stays uncommitted to the final production architecture. +**Parallel lightweight spike (2026-06-12)**: Specified in [008](../008-local-tts-lenovo-go-spike/plan.md); implemented in [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent). ALSA + Parakeet CPU + Piper + HID LED; brain is `get_agent()`. Not `riva.client` as the live ASR. Uncommitted to this plan's Riva/NIM primary path. **Serving Flexibility**: The LLM serving layer must be swappable. For v1, the target implementation uses the official NVIDIA NIM container (`nemotron-3-super-120b-a12b`) exposing an OpenAI-compatible endpoint at `http://localhost:8000/v1`. The agent is intended to use the existing `openai_compatible` provider for this deployment path. Future phases target clustered multi-node serving for full 340b-class models without changing the voice or agent code. diff --git a/specs/001-voice-dgx-spark-agent/spec.md b/specs/001-voice-dgx-spark-agent/spec.md index c3350d5..90fd288 100644 --- a/specs/001-voice-dgx-spark-agent/spec.md +++ b/specs/001-voice-dgx-spark-agent/spec.md @@ -130,7 +130,7 @@ We will use NeMo models (e.g., Conformer / FastConformer for ASR, FastPitch + Hi This decision was confirmed during spec review. -**Spike / Prototype Vehicle note (2026-06-12)**: A working local voice I/O spike exists in `conversational-voice-agent` using the Lenovo Go Wired Speaker as the physical interface (mic + speaker + Teams button for trigger + LED for session feedback). It provides a rapid, low-commitment way to prototype the natural-language voice feel (named-pipe or button trigger, live partial transcripts via streaming NeMo ASR, spoken replies on the device, hardware LED/button feedback) while feeding the *existing* agent + Supermemory harness we already have. The seams are intentionally minimal (`send_partial_to_agent` for input to the brain, `speak()` / `/tmp/voice_speak` for output audio on the Lenovo Go) so the spike does not re-implement or bypass the agent brain / Supermemory. See `specs/001-interim-lenovo-go-voice-spike.md` in that repo. This is a prototyping vehicle and is intentionally uncommitted to final production voice layer choices (Riva vs. other, telephony integration, heavy models, the main VoiceOrchestrator path, etc.). +**Spike / Prototype Vehicle note (2026-06-12)**: The interim Lenovo Go hardware loop is specified in [008-local-tts-lenovo-go-spike](../008-local-tts-lenovo-go-spike/spec.md) and executed in [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent). Live path is ALSA + Parakeet (NeMo CPU) + Piper, calling `get_agent()` in this package. It is uncommitted to the Riva / telephony / NIM compose choices in this spec. ## Related Work diff --git a/specs/001-voice-dgx-spark-agent/tasks.md b/specs/001-voice-dgx-spark-agent/tasks.md index 4e67c55..79d1e8a 100644 --- a/specs/001-voice-dgx-spark-agent/tasks.md +++ b/specs/001-voice-dgx-spark-agent/tasks.md @@ -19,7 +19,7 @@ This document breaks the plan into dependency-ordered, actionable tasks. Each ta **Current Status (as of 2026-06-12)**: Core agent + openai_compatible wiring + lazy voice imports complete and committed. `local-tts` 30B NeuTTS service initialized in dedicated repo with proper hygiene. nv-monitor observability tool integrated. Compose defaults to local-tts service. -A lightweight **spike** in `conversational-voice-agent` (Lenovo Go hardware loop) has real streaming NeMo ASR (riva.client, live partials), Teams LED feedback (active during session), and partials wired to the agent seam. The spike adds a `speak()` + speak-pipe reply path and wires the physical Teams button to the (locked) trigger pipe so E2E (pipe/button → live partial STT → existing agent + Supermemory via seams → spoken reply on device) can be prototyped and "felt" on real hardware. See the local-tts handoff + its focused spike spec and the light note in this spec.md. The spike deliberately uses a lighter brain path (no 120B lockup) and does not re-implement the agent harness / Supermemory. +The Lenovo Go spike checklist is [008/tasks.md](../008-local-tts-lenovo-go-spike/tasks.md). Code: [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent). Live ASR is Parakeet via NeMo on CPU, not `riva.client`. Riva compatibility on current hardware and full production voice loop (telephony / main orchestrator) remain future items. @@ -44,7 +44,9 @@ Riva compatibility on current hardware and full production voice loop (telephony - [ ] Clean up related documentation, compose files, and old scripts - [ ] Ensure all code changes are clean, well-tested where possible, and follow project conventions -**Spike Track – local-tts Lenovo Go Voice Loop (interim hardware prototype for natural interface feel)** (added 2026-06-12; exercises the *existing* agent + Supermemory via minimal seams; deliberately lightweight and uncommitted to production choices): +**Spike Track – moved to spec 008.** Do not treat the checkboxes below as the live ASR story (`riva.client` was an early note; Parakeet CPU shipped). Canonical list: [008/tasks.md](../008-local-tts-lenovo-go-spike/tasks.md). + +**Spike Track (historical 2026-06-12 notes, superseded by 008)**: - [x] Teams button light feedback implemented (led_control.py + integration in voice_loop; light stays active while session runs). - [x] Real streaming NeMo ASR integrated using riva.client (replaced mock; partial transcripts flow in real time from first chunk). - [x] Agent callback wiring: partial results sent to `send_partial_to_agent()` as soon as they arrive. diff --git a/specs/008-local-tts-lenovo-go-spike/plan.md b/specs/008-local-tts-lenovo-go-spike/plan.md new file mode 100644 index 0000000..5ed2b74 --- /dev/null +++ b/specs/008-local-tts-lenovo-go-spike/plan.md @@ -0,0 +1,87 @@ +# Plan: Local-tts Lenovo Go voice I/O spike (008) + +**Feature**: 008-local-tts-lenovo-go-spike +**Spec**: [spec.md](./spec.md) +**Date**: 2026-06-12 (work); recorded 2026-08-20 + +## 1. Architecture + +Two processes, two repos. This package is a library the I/O process imports. + +``` +Teams button (BTN_0) Lenovo Go speaker (ALSA) + │ ▲ + ▼ │ + button_listener ──pipe──► voice_loop ─┤ + arecord → VAD → Parakeet (CPU) + text ──► get_agent() [this repo] + reply ─► Piper → aplay (retry / settle) + LED via hidraw + telemetry JSONL (+ optional OTLP) +``` + +| Piece | Owner | +|-------|--------| +| Button, LED, ALSA, VAD, STT, TTS, telemetry | [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent) (working name **local-tts**) | +| LangGraph, memory injection, provider factory | **this repo** (`thelab_langchain.agent.graph.get_agent`) | + +`src/thelab_langchain/voice/` in *this* tree (Riva wrappers, streaming `NotImplementedError`) is **not** the live path for this spike. + +## 2. Tech choices (locked for the spike) + +| Concern | Choice | Why | +|---------|--------|-----| +| STT | Parakeet TDT 0.6B v3, NeMo, CPU | Fits Spark; no Riva GB10 image as a blocker | +| TTS | Piper `en_US-amy-medium`, then espeak-ng, then tone | Native aarch64; 30×-class CPU synth is a claim to *measure*, not to print unmeasured | +| Brain | Existing `get_agent()` | Do not fork memory/tools | +| LLM | Grok default; `make ollama` / `openai_compatible` for local | Spec 007: no 120B+ agent loops | +| Trigger | FIFO `/tmp/voice_trigger` | Same seam for button and `echo start` | +| Audio | ALSA `arecord`/`aplay`, name-based card discovery | USB card index moves on replug | +| VAD | RMS energy, not neural VAD | Enough for push-to-talk sessions | +| Interrupt | `stop_event` polled in playback; later also cancels agent wait | Button, not barge-in | +| Package layout | Separate git repo + isolated `.venv` | Matches basement-lab hygiene | + +## 3. Phases + +### Phase 0 — Brain seam (this repo) + +- `get_agent()` importable via `pip install -e`. +- `.env.example` documents `LLM_PROVIDER`, `XAI_API_KEY`, `SUPERMEMORY_API_KEY`. +- `make` here only **preps** the package; it does not start ALSA. + +### Phase 1 — Hardware loop (I/O package) + +- Record → VAD → STT → agent → TTS on the Go. +- LED + Teams button + settle/retry for half-duplex. +- Smoke path with no keys. + +### Phase 2 — Robustness (I/O package) + +- USB hotplug re-discovery. +- Session transcripts on-host only. +- Content-free OTEL (opt-in). +- Sentence-chunked TTS + EOU / TTFA metrics (blueprint-informed, Option A). + +Phase 2 is refinement of **this** spike, not a jump to spec 001 production voice. + +## 4. Risks + +| Risk | Mitigation | +|------|------------| +| USB “device busy” | Settle delay + `_robust_aplay` retries; `make audio-reset` | +| SIGINT swallowed while idle | Timed `Event.wait` + SIGINT/SIGTERM handlers (found after first public sessions) | +| Cross-repo drift | Document `get_agent()` as the only brain import; I/O does not copy graph.py | +| Over-claiming Riva/NIM | Keep this spec explicit: live path is ALSA + Parakeet + Piper | +| Secrets / serials in git | `.env` gitignored; no hardware serials in SDD or I/O git | + +## 5. Success metrics + +- One spoken session of several turns on the desk hardware. +- Smoke without this package. +- Per-turn latency table from real `turn_complete` events (n may be small; say so). +- Button interrupt of TTS. +- Hotplug recovery observed at least once. + +## 6. What this plan is not + +It is not a rewrite of spec 001. It is not a Hermes operating manual. Hermes wiki pages that describe `voice_loop.py` stay under `~/.hermes/wikis/local-tts/` and are not copied here. diff --git a/specs/008-local-tts-lenovo-go-spike/spec.md b/specs/008-local-tts-lenovo-go-spike/spec.md new file mode 100644 index 0000000..cfb9e9b --- /dev/null +++ b/specs/008-local-tts-lenovo-go-spike/spec.md @@ -0,0 +1,130 @@ +# Feature Spec: Local-tts Lenovo Go voice I/O spike + +**Feature ID**: 008-local-tts-lenovo-go-spike +**Status**: Specified and executed. +**Implementation**: [`derekclair/conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent) +(historically the `local-tts` package). This repo stays the brain (`get_agent()`). +**Created**: 2026-06-12 (spike locked) +**Recorded here**: 2026-08-20 +**Owner**: Derek Clair +**Parent**: [001-voice-dgx-spark-agent](../001-voice-dgx-spark-agent/spec.md) + +## Record-keeping note + +This spec is written **after** the spike was already running. The design was real; the SDD trail was not. Notes lived in a Hermes-generated wiki (`~/.hermes/wikis/local-tts/`, left in place — Hermes expects it) and in ad-hoc markdown in the I/O tree. This folder is the SDD record in the **brain** repo: what we asked the hardware loop to do, and what this package must expose so that loop does not re-implement the agent. + +It does **not** replace spec 001. Spec 001 is the broader desktop-voice goal (Riva/NIM compose, streaming barge-in, larger local models). This spec is the **interim hardware spike**: get a natural button → listen → think → speak loop on a Lenovo Go attached to a DGX Spark, using the **existing** LangGraph brain. + +## Overview + +Build a small, isolated voice I/O process for the Lenovo Go Wired Speaker: + +1. Teams button starts or ends a multi-turn session (LED on while the session is live). +2. Energy-based VAD captures an utterance from the Go microphone. +3. Local STT (NVIDIA Parakeet via NeMo, CPU path on this Spark) turns audio into text. +4. This repo’s `get_agent()` runs the turn (Supermemory injection + LLM + optional tools). +5. Local TTS speaks the reply on the same USB device (Piper primary; espeak-ng then a diagnostic tone as fallback). +6. The Go is half-duplex: capture and playback must not overlap; “device busy” must recover. + +The spike owns **ears, mouth, hands, and telemetry**. This package owns **brain and memory**. Seams stay thin on purpose. + +## Goals + +- Prove the desk hardware can drive a real multi-turn conversation with the existing agent. +- Keep STT/TTS on-device. Reasoning may use Grok or a local OpenAI-compatible endpoint. +- Do not rebuild LangGraph, Supermemory, or provider routing in the I/O process. +- Survive USB unplug/replug without a full restart of the button path. +- Measure per-turn latency (ASR / agent / TTS / later EOU and time-to-first-audio) without shipping transcripts off-box. + +## Non-goals + +- Telephony, hosted STT/TTS, or a browser/WebRTC UI. +- Full Nemotron Voice Agent NIM compose as the primary desk UX. +- Voice barge-in (talking over the agent). Button interrupt of playback is in scope; speech-over-speech is not. +- Replacing `thelab_langchain.voice` Riva helpers as the live path (those remain a Phase-2 experiment in this tree). +- Multi-tenant product; unpublished-company / customer-service framing. + +## User stories + +1. As the person at the desk, I press the Teams button, hear “Ready.”, speak, and hear a spoken reply through the same speakerphone. +2. As that person, I press the button again to stop speech mid-utterance and end the session. +3. As a developer, I run a smoke path (LED + TTS) with **no** API keys and **no** this package installed. +4. As a developer, I install this package editable into the I/O venv and get the real agent + memory path. +5. As an operator, I unplug and replug the Go and the button/LED/ALSA card rebind without a manual service restart. + +## Functional requirements + +### FR-1 Trigger and session + +- Named pipe `/tmp/voice_trigger` is the session seam (button listener writes `start`; the loop can also be kicked with `echo start > /tmp/voice_trigger`). +- First press starts a session; press during a session sets a stop event (cancel in-flight TTS; abandon the in-flight agent result when that is wired). +- LED (HID report on the Go) is solid while services are ready / a session is active; a short blink marks session end. + +### FR-2 Capture and STT + +- Record from the Go ALSA device (discover by name, not a frozen card index). +- End-of-utterance by energy VAD (configurable silence window; default on the order of 0.5 s once tuned). +- Transcribe with Parakeet TDT 0.6B v3 via NeMo on the CPU path for this Spark. +- Empty transcription is a spoken apology, not a crash. + +### FR-3 Agent seam (this repo) + +- I/O process calls `thelab_langchain.agent.graph.get_agent(user_id)` and `graph.invoke(...)`. +- Caller accumulates `HumanMessage` / `AIMessage` for the session (`thread_id` per session). +- No second copy of memory tools or provider factory in the I/O process. +- Missing keys or missing package → mock/fallback path so hardware bring-up can continue. + +### FR-4 TTS and half-duplex audio + +- Primary: Piper neural TTS. Fallback: espeak-ng, then a diagnostic tone. +- Retry `aplay` on “device busy” with a short settle between capture and playback. +- Stop event must terminate playback promptly. +- Later: sentence-chunked Piper so first audio does not wait on the full reply (time-to-first-audio). + +### FR-5 Hotplug + +- Button listener remains active if the Go is missing at start or is yanked. +- Re-discover evdev (Teams `BTN_0`, smallest keyset), ALSA card index, and hidraw LED device on udev add/remove. + +### FR-6 Observability + +- Local structured events (JSONL). Per-turn durations: `asr_ms`, `agent_ms`, `tts_ms`, `total_ms`, and later `eou_ms` / `tts_ttfa_ms`. +- Optional OTLP export is **opt-in** and **content-free** (durations and counts only). No transcripts on the wire. +- Transcripts, if kept, stay on the host and are gitignored. + +## Non-functional requirements + +- Python 3.11, project-local `.venv`, no global pip. +- aarch64 DGX Spark (CUDA present; do not require `nvidia-smi` or `torch.cuda.is_available()`). +- User in `audio` and `input`; udev so hidraw is writable without daily sudo. +- MIT; no secrets in git; no hardware serials in git. + +## Acceptance criteria + +- [ ] `make smoke` in the I/O package runs LED + TTS with no this-package import and no API keys. +- [ ] With this package editable-installed and keys present, a button-started session completes at least one STT → `get_agent()` → TTS turn. +- [ ] Half-duplex: a turn that records then plays does not stick on “device busy” without a retry/settle path. +- [ ] Button press during TTS stops playback. +- [ ] Unplug/replug recovers button + LED + ALSA without restarting the listener process. +- [ ] A `turn_complete` event includes numeric stage latencies. Transcripts are not required on any remote collector. + +## Seams this package must keep stable + +| Seam | Contract | +|------|----------| +| `get_agent(user_id)` | Returns a compiled LangGraph. No checkpointer required for the spike. | +| `graph.invoke({"messages": ...})` | Accepts accumulated session messages. | +| `.env` / `LLM_*` | Same provider factory as `thelab-chat`. | +| Editable install | `pip install -e ` from the I/O venv. | + +## Relationship to other specs + +- **001** — long-term voice agent (Riva/NIM/streaming). This spike is explicitly uncommitted to those choices. +- **007** — Spark memory budget. STT/TTS stay on CPU so the GPU slot can stay with a ~30B worker or stay free. +- I/O-repo specs (USB hotplug, session transcripts, blueprint-informed latency) refine *this* spike; they do not replace it. + +## Open questions (left to the I/O repo) + +- Exact Piper voice file location and fallback order. +- Whether OTEL goes to a LAN hub or stays JSONL-only. +- Whether to extract a modular pipeline later (spec 001 / blueprint Option B). Not required to call the spike done. diff --git a/specs/008-local-tts-lenovo-go-spike/tasks.md b/specs/008-local-tts-lenovo-go-spike/tasks.md new file mode 100644 index 0000000..57e3e37 --- /dev/null +++ b/specs/008-local-tts-lenovo-go-spike/tasks.md @@ -0,0 +1,44 @@ +# Tasks: Local-tts Lenovo Go voice I/O spike (008) + +**Feature**: 008-local-tts-lenovo-go-spike +**Spec**: [spec.md](./spec.md) · **Plan**: [plan.md](./plan.md) + +Checkboxes record what the spike actually did. This file was filled in when +the SDD record was added to this repo, not on the original 2026-06-12 day. + +## Phase 0 — Brain seam (this repo) + +- [x] Keep `get_agent()` as the public graph factory +- [x] Provider factory (`xai` / `anthropic` / `openai_compatible`) +- [x] `.env` gitignored; `.env.example` for keys +- [x] `make` / `make local` only preps the editable package +- [ ] LangGraph checkpointer for voice sessions (spec 004 — not required to call 008 done) + +## Phase 1 — Hardware loop (I/O package, working name local-tts) + +- [x] Named pipe trigger `/tmp/voice_trigger` +- [x] Teams button → pipe +- [x] HID LED on session / ready +- [x] Energy VAD record on the Go capture device +- [x] Parakeet TDT 0.6B v3 CPU STT +- [x] `get_agent()` + session message list +- [x] Piper TTS with espeak-ng / tone fallback +- [x] Half-duplex settle + aplay retry +- [x] `make smoke` without this package / without keys +- [x] Volume dial ticks (nice-to-have; shipped) + +## Phase 2 — Robustness (same I/O package) + +- [x] USB hotplug re-discovery (evdev + ALSA + hidraw) +- [x] On-host session transcripts (gitignored) +- [x] Content-free opt-in OTEL +- [x] Sentence-chunked Piper + `eou_ms` / `tts_ttfa_ms` +- [x] SIGINT/SIGTERM actually stop the idle loop +- [ ] Voice barge-in (talk over TTS) — out of scope; button cancel only +- [ ] Riva streaming ASR as primary — out of scope for 008 + +## Traceability + +Implementation git history for Phase 1–2 is +[`derekclair/conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent). +This tasks file is only the checklist view. diff --git a/specs/README.md b/specs/README.md index 9d9ca99..5a3e00b 100644 --- a/specs/README.md +++ b/specs/README.md @@ -12,6 +12,7 @@ are drafts or future work, not a claim that every spec is implemented. | [005](005-testing-and-cicd/spec.md) | Testing & CI | Direction; a CPU-only pytest + ruff workflow is in `.github/workflows/ci.yml` | | [006](006-alternative-memory-systems/spec.md) | Alternative memory backends | Future consideration; Supermemory is the current store | | [007](007-dgx-hardware-optimization/spec.md) | DGX Spark hardware budget | ~30B-class local models; no 120B+ agent loops on one Spark | +| [008](008-local-tts-lenovo-go-spike/spec.md) | Local-tts Lenovo Go spike | Button → VAD → Parakeet → `get_agent()` → Piper. **Executed in** [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent) | Workstation **fleet operations** (orchestrator / architect / researcher / coder / reviewer, Kanban vs chat) are not specified here. That operating