A voice AI agent for a hardware store, built with LiveKit Agents and LiveKit Cloud.
The agent continuously monitors the caller's emotional state using Hume AI voice prosody analysis. When high-intensity negative emotions like anger or frustration are detected, the system automatically escalates the call to a human agent without requiring the customer to ask. This happens seamlessly in the background while the conversation continues naturally.
When escalation is triggered (either by emotion detection or customer request), the agent performs a warm transfer via SIP trunk. The caller is placed on a brief hold while the system connects to a supervisor, passing along conversation context and the reason for escalation. The AI agent then gracefully exits, handing the call to the human.
When a customer describes a DIY project ("I want to build a raised garden bed"), the agent uses a LangGraph workflow to:
- Clarify requirements - If the project description is ambiguous (missing size, scope unclear), the workflow interrupts to ask the customer for clarification
- Identify materials - Uses Google Gemini to extract specific materials needed (lumber dimensions, screws, soil, etc.)
- Semantic product search - Matches materials to actual inventory using FAISS vector search with Gemini embeddings
- Check availability - Verifies stock at the customer's preferred location
- Cross-location fallback - If items are out of stock, checks other store locations
- Generate shopping list - Returns a complete list with prices, aisle locations, and total estimate
This demonstrates a multi-step agentic workflow where an LLM orchestrates multiple tools to accomplish a complex task.
The project assistant is implemented as an independent LangGraph workflow that integrates with the voice agent via a tool interface. This architecture provides several benefits:
Testable in isolation - The ProjectAssistantWorkflow can be unit tested independently of the voice agent. Each node (analyze_project, identify_materials, match_products, etc.) can be tested with mock data.
Composable - The voice agent calls the workflow as a tool. You can add other specialized workflows (e.g., ReturnAssistantWorkflow, WarrantyLookupWorkflow) following the same pattern.
Clear separation of concerns - The workflow handles its own:
- State management (
ProjectAssistantState) - LLM interactions (Gemini for analysis)
- Data layer (FAISS vector store, inventory)
While the voice agent handles:
- Speech and conversation flow
- Tool orchestration
- User interaction
Independent human-in-the-loop - The LangGraph interrupt/resume pattern works at the workflow level. The voice agent simply handles the needs_clarification response and calls back with the customer's answer.
┌─────────────────────────────────────────────────────────────┐
│ Voice Agent (LiveKit) │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ STT/TTS │ │ Turn │ │ Tools │ │
│ │ Pipeline │ │ Detection │ │ - inventory_check │ │
│ │ │ │ │ │ - store_hours │ │
│ │ │ │ │ │ - project_assistant─┼──┼──┐
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │
└─────────────────────────────────────────────────────────────┘ │
│
┌─────────────────────────────────────────────────────────────┐ │
│ Project Assistant (LangGraph) │◄─┘
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────────┐ │
│ │ Analyze │─►│ Clarify │─►│ Identify │─►│ Match │ │
│ │ Project │ │ (HITL) │ │ Materials│ │ Products │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────┬──────┘ │
│ │ │
│ ┌──────────────────┐ ┌───────────────────────┐ │ │
│ │ Generate │◄─│ Check Availability │◄─┘ │
│ │ Shopping List │ │ │ │
│ └──────────────────┘ └───────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Data Layer: FAISS Vector Store + Inventory JSON │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
This pattern scales well - each "assistant" can be developed, tested, and maintained independently, then integrated via a simple tool interface on the main agent.
The agent has access to real-time store data through callable tools:
- Inventory lookup - Check product availability, pricing, and aisle location at specific stores
- Project assistant - Help customers plan DIY projects with complete shopping lists
- Store hours - Get operating hours for each location
- Department info - List available departments (Sales, Tool Rental, Pro Desk, etc.)
The agent automatically responds in the caller's language. Using ElevenLabs Scribe v2 for speech-to-text (99+ languages), Google Gemini for understanding and response generation, and Cartesia for text-to-speech, language detection and response happens automatically with no manual selection needed.
- LiveKit Turn Detector for contextually-aware end-of-turn detection with multilingual support
- Background voice cancellation optimized for telephony
- Preemptive response generation for lower latency
- Twilio SIP integration for inbound phone calls
This project is designed to work with Claude Code.
To get the most out of Claude Code, install the LiveKit Docs MCP server:
claude mcp add --transport http livekit-docs https://docs.livekit.io/mcp
The project includes a complete AGENTS.md file. To learn more about this file, see https://agents.md.
Clone the repository and install dependencies to a virtual environment:
git clone https://github.com/yorrickjansen/livekit-hardware-store-agent.git
cd livekit-hardware-store-agent
uv syncSign up for LiveKit Cloud then set up the environment by copying .env.example to .env.local and filling in the required keys:
LIVEKIT_URLLIVEKIT_API_KEYLIVEKIT_API_SECRET
You can load the LiveKit environment automatically using the LiveKit CLI:
lk cloud auth
lk app env -w -d .env.localTo enable the project-based shopping assistant, add your Google Gemini API key to .env.local:
GEMINI_API_KEY=your-api-key-here
Get your API key at Google AI Studio.
Then generate the product embeddings (one-time setup):
uv run python scripts/generate_embeddings.pyThis creates vector embeddings for ~300 products across 3 store locations, enabling semantic search for the project assistant.
Before your first run, you must download certain models such as Silero VAD and the LiveKit turn detector:
uv run python src/agent.py download-filesNext, run this command to speak to your agent directly in your terminal:
uv run python src/agent.py consoleTo run the agent for use with a frontend or telephony, use the dev command:
uv run python src/agent.py devIn production, use the start command:
uv run python src/agent.py startTo enable phone calls to your agent via Twilio, follow these steps:
- A Twilio account with a purchased phone number
- Twilio CLI installed and configured
- LiveKit CLI installed and authenticated
twilio api trunking v1 trunks create \
--friendly-name "My LiveKit Trunk" \
--domain-name "my-trunk.pstn.twilio.com"Save the Trunk SID (e.g., TK...) from the output.
Get your LiveKit SIP URI from the LiveKit Cloud dashboard under Telephony → SIP trunks. It looks like sip:abc123.sip.livekit.cloud.
Important: Use the SIP URI shown in the dashboard, not your project subdomain.
twilio api trunking v1 trunks origination-urls create \
--trunk-sid <your-trunk-sid> \
--friendly-name "LiveKit SIP URI" \
--sip-url "sip:<your-sip-uri>.sip.livekit.cloud" \
--weight 1 --priority 1 --enabled# List your phone numbers to get the SID
twilio phone-numbers list
# Associate the phone number with the trunk
twilio api trunking v1 trunks phone-numbers create \
--trunk-sid <your-trunk-sid> \
--phone-number-sid <your-phone-number-sid>lk sip inbound create --name "Twilio Inbound" --numbers "+1234567890"Replace +1234567890 with your Twilio phone number.
lk sip dispatch create --name "Inbound Calls" --individual "call-" --randomizelk agent deployAfter completing these steps, call your Twilio phone number. The call should connect to your LiveKit agent.
To troubleshoot:
- Check agent status:
lk agent status - View agent logs:
lk agent logs - List rooms:
lk room list - Check LiveKit Cloud Telephony → Calls for call logs
This project includes a complete suite of evals, based on the LiveKit Agents testing & evaluation framework. To run them, use pytest.
uv run pytestThis project is licensed under the MIT License - see the LICENSE file for details.