Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@ What is actually in the tree:
- Specs for the voice/desktop goal, Spark memory budget, and follow-on work
(`specs/`)

> **Used by:** the companion [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent) repo — a local voice front-end (STT/TTS on a DGX Spark) that uses this package as its agent brain.
> **Used by:** [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent)
> executes [spec 008](specs/008-local-tts-lenovo-go-spike/spec.md) (the Lenovo Go
> local-tts spike). This package is `get_agent()` only.
>
> **Fleet ops:** the workstation's multi-profile fleet (orchestrator, architect,
> researcher, coder, reviewer) is documented in local Hermes docs at
Expand Down Expand Up @@ -160,6 +162,7 @@ examples/ Non-interactive snippets
- **[Architecture](docs/architecture.md)** — layering of this package
- **[Development](docs/development.md)** — venv, chat, common commands
- **[Specs index](specs/README.md)** — design and planning already in this repo
- **[Spec 008](specs/008-local-tts-lenovo-go-spike/spec.md)** — local-tts spike; implemented in [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent)
- Workstation fleet operating manual (Hermes, not vendored):
`~/.hermes/docs/agentic-workflow.md`

Expand Down
2 changes: 1 addition & 1 deletion specs/001-voice-dgx-spark-agent/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ We will build a **layered voice agent** with clear separation of concerns:

**Key Principle**: The voice layer is "dumb but fast". The LangGraph agent is the intelligent brain that decides *what* to say and *when* to use memory/tools.

**Parallel lightweight spike (2026-06-12)**: A low-commitment hardware prototype loop lives in `conversational-voice-agent`: named-pipe (or Teams button) trigger, direct ALSA arecord/aplay on the specific Lenovo Go devices, `riva.client` streaming ASR with real partials, raw HID for the Teams LED, and a `speak()` / speak-pipe entry point for replies. It is used for rapid prototyping of the natural voice "feel" on real desk hardware and for exercising the *existing* agent + Supermemory (via the minimal seams) while the primary `thelab_langchain.voice` (orchestrator + audio + VAD) and telephony production path mature. See the local-tts handoff and its spike spec. The spike deliberately stays uncommitted to the final production architecture.
**Parallel lightweight spike (2026-06-12)**: Specified in [008](../008-local-tts-lenovo-go-spike/plan.md); implemented in [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent). ALSA + Parakeet CPU + Piper + HID LED; brain is `get_agent()`. Not `riva.client` as the live ASR. Uncommitted to this plan's Riva/NIM primary path.

**Serving Flexibility**: The LLM serving layer must be swappable. For v1, the target implementation uses the official NVIDIA NIM container (`nemotron-3-super-120b-a12b`) exposing an OpenAI-compatible endpoint at `http://localhost:8000/v1`. The agent is intended to use the existing `openai_compatible` provider for this deployment path. Future phases target clustered multi-node serving for full 340b-class models without changing the voice or agent code.

Expand Down
2 changes: 1 addition & 1 deletion specs/001-voice-dgx-spark-agent/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ We will use NeMo models (e.g., Conformer / FastConformer for ASR, FastPitch + Hi

This decision was confirmed during spec review.

**Spike / Prototype Vehicle note (2026-06-12)**: A working local voice I/O spike exists in `conversational-voice-agent` using the Lenovo Go Wired Speaker as the physical interface (mic + speaker + Teams button for trigger + LED for session feedback). It provides a rapid, low-commitment way to prototype the natural-language voice feel (named-pipe or button trigger, live partial transcripts via streaming NeMo ASR, spoken replies on the device, hardware LED/button feedback) while feeding the *existing* agent + Supermemory harness we already have. The seams are intentionally minimal (`send_partial_to_agent` for input to the brain, `speak()` / `/tmp/voice_speak` for output audio on the Lenovo Go) so the spike does not re-implement or bypass the agent brain / Supermemory. See `specs/001-interim-lenovo-go-voice-spike.md` in that repo. This is a prototyping vehicle and is intentionally uncommitted to final production voice layer choices (Riva vs. other, telephony integration, heavy models, the main VoiceOrchestrator path, etc.).
**Spike / Prototype Vehicle note (2026-06-12)**: The interim Lenovo Go hardware loop is specified in [008-local-tts-lenovo-go-spike](../008-local-tts-lenovo-go-spike/spec.md) and executed in [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent). Live path is ALSA + Parakeet (NeMo CPU) + Piper, calling `get_agent()` in this package. It is uncommitted to the Riva / telephony / NIM compose choices in this spec.

## Related Work

Expand Down
6 changes: 4 additions & 2 deletions specs/001-voice-dgx-spark-agent/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This document breaks the plan into dependency-ordered, actionable tasks. Each ta

**Current Status (as of 2026-06-12)**: Core agent + openai_compatible wiring + lazy voice imports complete and committed. `local-tts` 30B NeuTTS service initialized in dedicated repo with proper hygiene. nv-monitor observability tool integrated. Compose defaults to local-tts service.

A lightweight **spike** in `conversational-voice-agent` (Lenovo Go hardware loop) has real streaming NeMo ASR (riva.client, live partials), Teams LED feedback (active during session), and partials wired to the agent seam. The spike adds a `speak()` + speak-pipe reply path and wires the physical Teams button to the (locked) trigger pipe so E2E (pipe/button → live partial STT → existing agent + Supermemory via seams → spoken reply on device) can be prototyped and "felt" on real hardware. See the local-tts handoff + its focused spike spec and the light note in this spec.md. The spike deliberately uses a lighter brain path (no 120B lockup) and does not re-implement the agent harness / Supermemory.
The Lenovo Go spike checklist is [008/tasks.md](../008-local-tts-lenovo-go-spike/tasks.md). Code: [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent). Live ASR is Parakeet via NeMo on CPU, not `riva.client`.

Riva compatibility on current hardware and full production voice loop (telephony / main orchestrator) remain future items.

Expand All @@ -44,7 +44,9 @@ Riva compatibility on current hardware and full production voice loop (telephony
- [ ] Clean up related documentation, compose files, and old scripts
- [ ] Ensure all code changes are clean, well-tested where possible, and follow project conventions

**Spike Track – local-tts Lenovo Go Voice Loop (interim hardware prototype for natural interface feel)** (added 2026-06-12; exercises the *existing* agent + Supermemory via minimal seams; deliberately lightweight and uncommitted to production choices):
**Spike Track – moved to spec 008.** Do not treat the checkboxes below as the live ASR story (`riva.client` was an early note; Parakeet CPU shipped). Canonical list: [008/tasks.md](../008-local-tts-lenovo-go-spike/tasks.md).

**Spike Track (historical 2026-06-12 notes, superseded by 008)**:
- [x] Teams button light feedback implemented (led_control.py + integration in voice_loop; light stays active while session runs).
- [x] Real streaming NeMo ASR integrated using riva.client (replaced mock; partial transcripts flow in real time from first chunk).
- [x] Agent callback wiring: partial results sent to `send_partial_to_agent()` as soon as they arrive.
Expand Down
87 changes: 87 additions & 0 deletions specs/008-local-tts-lenovo-go-spike/plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# Plan: Local-tts Lenovo Go voice I/O spike (008)

**Feature**: 008-local-tts-lenovo-go-spike
**Spec**: [spec.md](./spec.md)
**Date**: 2026-06-12 (work); recorded 2026-08-20

## 1. Architecture

Two processes, two repos. This package is a library the I/O process imports.

```
Teams button (BTN_0) Lenovo Go speaker (ALSA)
│ ▲
▼ │
button_listener ──pipe──► voice_loop ─┤
arecord → VAD → Parakeet (CPU)
text ──► get_agent() [this repo]
reply ─► Piper → aplay (retry / settle)
LED via hidraw
telemetry JSONL (+ optional OTLP)
```

| Piece | Owner |
|-------|--------|
| Button, LED, ALSA, VAD, STT, TTS, telemetry | [`conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent) (working name **local-tts**) |
| LangGraph, memory injection, provider factory | **this repo** (`thelab_langchain.agent.graph.get_agent`) |

`src/thelab_langchain/voice/` in *this* tree (Riva wrappers, streaming `NotImplementedError`) is **not** the live path for this spike.

## 2. Tech choices (locked for the spike)

| Concern | Choice | Why |
|---------|--------|-----|
| STT | Parakeet TDT 0.6B v3, NeMo, CPU | Fits Spark; no Riva GB10 image as a blocker |
| TTS | Piper `en_US-amy-medium`, then espeak-ng, then tone | Native aarch64; 30×-class CPU synth is a claim to *measure*, not to print unmeasured |
| Brain | Existing `get_agent()` | Do not fork memory/tools |
| LLM | Grok default; `make ollama` / `openai_compatible` for local | Spec 007: no 120B+ agent loops |
| Trigger | FIFO `/tmp/voice_trigger` | Same seam for button and `echo start` |
| Audio | ALSA `arecord`/`aplay`, name-based card discovery | USB card index moves on replug |
| VAD | RMS energy, not neural VAD | Enough for push-to-talk sessions |
| Interrupt | `stop_event` polled in playback; later also cancels agent wait | Button, not barge-in |
| Package layout | Separate git repo + isolated `.venv` | Matches basement-lab hygiene |

## 3. Phases

### Phase 0 — Brain seam (this repo)

- `get_agent()` importable via `pip install -e`.
- `.env.example` documents `LLM_PROVIDER`, `XAI_API_KEY`, `SUPERMEMORY_API_KEY`.
- `make` here only **preps** the package; it does not start ALSA.

### Phase 1 — Hardware loop (I/O package)

- Record → VAD → STT → agent → TTS on the Go.
- LED + Teams button + settle/retry for half-duplex.
- Smoke path with no keys.

### Phase 2 — Robustness (I/O package)

- USB hotplug re-discovery.
- Session transcripts on-host only.
- Content-free OTEL (opt-in).
- Sentence-chunked TTS + EOU / TTFA metrics (blueprint-informed, Option A).

Phase 2 is refinement of **this** spike, not a jump to spec 001 production voice.

## 4. Risks

| Risk | Mitigation |
|------|------------|
| USB “device busy” | Settle delay + `_robust_aplay` retries; `make audio-reset` |
| SIGINT swallowed while idle | Timed `Event.wait` + SIGINT/SIGTERM handlers (found after first public sessions) |
| Cross-repo drift | Document `get_agent()` as the only brain import; I/O does not copy graph.py |
| Over-claiming Riva/NIM | Keep this spec explicit: live path is ALSA + Parakeet + Piper |
| Secrets / serials in git | `.env` gitignored; no hardware serials in SDD or I/O git |

## 5. Success metrics

- One spoken session of several turns on the desk hardware.
- Smoke without this package.
- Per-turn latency table from real `turn_complete` events (n may be small; say so).
- Button interrupt of TTS.
- Hotplug recovery observed at least once.

## 6. What this plan is not

It is not a rewrite of spec 001. It is not a Hermes operating manual. Hermes wiki pages that describe `voice_loop.py` stay under `~/.hermes/wikis/local-tts/` and are not copied here.
130 changes: 130 additions & 0 deletions specs/008-local-tts-lenovo-go-spike/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# Feature Spec: Local-tts Lenovo Go voice I/O spike

**Feature ID**: 008-local-tts-lenovo-go-spike
**Status**: Specified and executed.
**Implementation**: [`derekclair/conversational-voice-agent`](https://github.com/derekclair/conversational-voice-agent)
(historically the `local-tts` package). This repo stays the brain (`get_agent()`).
**Created**: 2026-06-12 (spike locked)
**Recorded here**: 2026-08-20
**Owner**: Derek Clair
**Parent**: [001-voice-dgx-spark-agent](../001-voice-dgx-spark-agent/spec.md)

## Record-keeping note

This spec is written **after** the spike was already running. The design was real; the SDD trail was not. Notes lived in a Hermes-generated wiki (`~/.hermes/wikis/local-tts/`, left in place — Hermes expects it) and in ad-hoc markdown in the I/O tree. This folder is the SDD record in the **brain** repo: what we asked the hardware loop to do, and what this package must expose so that loop does not re-implement the agent.

It does **not** replace spec 001. Spec 001 is the broader desktop-voice goal (Riva/NIM compose, streaming barge-in, larger local models). This spec is the **interim hardware spike**: get a natural button → listen → think → speak loop on a Lenovo Go attached to a DGX Spark, using the **existing** LangGraph brain.

## Overview

Build a small, isolated voice I/O process for the Lenovo Go Wired Speaker:

1. Teams button starts or ends a multi-turn session (LED on while the session is live).
2. Energy-based VAD captures an utterance from the Go microphone.
3. Local STT (NVIDIA Parakeet via NeMo, CPU path on this Spark) turns audio into text.
4. This repo’s `get_agent()` runs the turn (Supermemory injection + LLM + optional tools).
5. Local TTS speaks the reply on the same USB device (Piper primary; espeak-ng then a diagnostic tone as fallback).
6. The Go is half-duplex: capture and playback must not overlap; “device busy” must recover.

The spike owns **ears, mouth, hands, and telemetry**. This package owns **brain and memory**. Seams stay thin on purpose.

## Goals

- Prove the desk hardware can drive a real multi-turn conversation with the existing agent.
- Keep STT/TTS on-device. Reasoning may use Grok or a local OpenAI-compatible endpoint.
- Do not rebuild LangGraph, Supermemory, or provider routing in the I/O process.
- Survive USB unplug/replug without a full restart of the button path.
- Measure per-turn latency (ASR / agent / TTS / later EOU and time-to-first-audio) without shipping transcripts off-box.

## Non-goals

- Telephony, hosted STT/TTS, or a browser/WebRTC UI.
- Full Nemotron Voice Agent NIM compose as the primary desk UX.
- Voice barge-in (talking over the agent). Button interrupt of playback is in scope; speech-over-speech is not.
- Replacing `thelab_langchain.voice` Riva helpers as the live path (those remain a Phase-2 experiment in this tree).
- Multi-tenant product; unpublished-company / customer-service framing.

## User stories

1. As the person at the desk, I press the Teams button, hear “Ready.”, speak, and hear a spoken reply through the same speakerphone.
2. As that person, I press the button again to stop speech mid-utterance and end the session.
3. As a developer, I run a smoke path (LED + TTS) with **no** API keys and **no** this package installed.
4. As a developer, I install this package editable into the I/O venv and get the real agent + memory path.
5. As an operator, I unplug and replug the Go and the button/LED/ALSA card rebind without a manual service restart.

## Functional requirements

### FR-1 Trigger and session

- Named pipe `/tmp/voice_trigger` is the session seam (button listener writes `start`; the loop can also be kicked with `echo start > /tmp/voice_trigger`).
- First press starts a session; press during a session sets a stop event (cancel in-flight TTS; abandon the in-flight agent result when that is wired).
- LED (HID report on the Go) is solid while services are ready / a session is active; a short blink marks session end.

### FR-2 Capture and STT

- Record from the Go ALSA device (discover by name, not a frozen card index).
- End-of-utterance by energy VAD (configurable silence window; default on the order of 0.5 s once tuned).
- Transcribe with Parakeet TDT 0.6B v3 via NeMo on the CPU path for this Spark.
- Empty transcription is a spoken apology, not a crash.

### FR-3 Agent seam (this repo)

- I/O process calls `thelab_langchain.agent.graph.get_agent(user_id)` and `graph.invoke(...)`.
- Caller accumulates `HumanMessage` / `AIMessage` for the session (`thread_id` per session).
- No second copy of memory tools or provider factory in the I/O process.
- Missing keys or missing package → mock/fallback path so hardware bring-up can continue.

### FR-4 TTS and half-duplex audio

- Primary: Piper neural TTS. Fallback: espeak-ng, then a diagnostic tone.
- Retry `aplay` on “device busy” with a short settle between capture and playback.
- Stop event must terminate playback promptly.
- Later: sentence-chunked Piper so first audio does not wait on the full reply (time-to-first-audio).

### FR-5 Hotplug

- Button listener remains active if the Go is missing at start or is yanked.
- Re-discover evdev (Teams `BTN_0`, smallest keyset), ALSA card index, and hidraw LED device on udev add/remove.

### FR-6 Observability

- Local structured events (JSONL). Per-turn durations: `asr_ms`, `agent_ms`, `tts_ms`, `total_ms`, and later `eou_ms` / `tts_ttfa_ms`.
- Optional OTLP export is **opt-in** and **content-free** (durations and counts only). No transcripts on the wire.
- Transcripts, if kept, stay on the host and are gitignored.

## Non-functional requirements

- Python 3.11, project-local `.venv`, no global pip.
- aarch64 DGX Spark (CUDA present; do not require `nvidia-smi` or `torch.cuda.is_available()`).
- User in `audio` and `input`; udev so hidraw is writable without daily sudo.
- MIT; no secrets in git; no hardware serials in git.

## Acceptance criteria

- [ ] `make smoke` in the I/O package runs LED + TTS with no this-package import and no API keys.
- [ ] With this package editable-installed and keys present, a button-started session completes at least one STT → `get_agent()` → TTS turn.
- [ ] Half-duplex: a turn that records then plays does not stick on “device busy” without a retry/settle path.
- [ ] Button press during TTS stops playback.
- [ ] Unplug/replug recovers button + LED + ALSA without restarting the listener process.
- [ ] A `turn_complete` event includes numeric stage latencies. Transcripts are not required on any remote collector.

## Seams this package must keep stable

| Seam | Contract |
|------|----------|
| `get_agent(user_id)` | Returns a compiled LangGraph. No checkpointer required for the spike. |
| `graph.invoke({"messages": ...})` | Accepts accumulated session messages. |
| `.env` / `LLM_*` | Same provider factory as `thelab-chat`. |
| Editable install | `pip install -e <this-repo>` from the I/O venv. |

## Relationship to other specs

- **001** — long-term voice agent (Riva/NIM/streaming). This spike is explicitly uncommitted to those choices.
- **007** — Spark memory budget. STT/TTS stay on CPU so the GPU slot can stay with a ~30B worker or stay free.
- I/O-repo specs (USB hotplug, session transcripts, blueprint-informed latency) refine *this* spike; they do not replace it.

## Open questions (left to the I/O repo)

- Exact Piper voice file location and fallback order.
- Whether OTEL goes to a LAN hub or stays JSONL-only.
- Whether to extract a modular pipeline later (spec 001 / blueprint Option B). Not required to call the spike done.
Loading
Loading