Skip to content

About

A voice AI agent for a hardware store, built with LiveKit Agents

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

LiveKit logo

Hardware Store Voice Agent

A voice AI agent for a hardware store, built with LiveKit Agents and LiveKit Cloud.

Features

Emotion-Based Call Escalation

The agent continuously monitors the caller's emotional state using Hume AI voice prosody analysis. When high-intensity negative emotions like anger or frustration are detected, the system automatically escalates the call to a human agent without requiring the customer to ask. This happens seamlessly in the background while the conversation continues naturally.

Warm Transfers to Human Agents

When escalation is triggered (either by emotion detection or customer request), the agent performs a warm transfer via SIP trunk. The caller is placed on a brief hold while the system connects to a supervisor, passing along conversation context and the reason for escalation. The AI agent then gracefully exits, handing the call to the human.

Project-Based Shopping Assistant (LangGraph)

When a customer describes a DIY project ("I want to build a raised garden bed"), the agent uses a LangGraph workflow to:

  1. Clarify requirements - If the project description is ambiguous (missing size, scope unclear), the workflow interrupts to ask the customer for clarification
  2. Identify materials - Uses Google Gemini to extract specific materials needed (lumber dimensions, screws, soil, etc.)
  3. Semantic product search - Matches materials to actual inventory using FAISS vector search with Gemini embeddings
  4. Check availability - Verifies stock at the customer's preferred location
  5. Cross-location fallback - If items are out of stock, checks other store locations
  6. Generate shopping list - Returns a complete list with prices, aisle locations, and total estimate

This demonstrates a multi-step agentic workflow where an LLM orchestrates multiple tools to accomplish a complex task.

Architecture

Modular Workflow Design

The project assistant is implemented as an independent LangGraph workflow that integrates with the voice agent via a tool interface. This architecture provides several benefits:

Testable in isolation - The ProjectAssistantWorkflow can be unit tested independently of the voice agent. Each node (analyze_project, identify_materials, match_products, etc.) can be tested with mock data.

Composable - The voice agent calls the workflow as a tool. You can add other specialized workflows (e.g., ReturnAssistantWorkflow, WarrantyLookupWorkflow) following the same pattern.

Clear separation of concerns - The workflow handles its own:

  • State management (ProjectAssistantState)
  • LLM interactions (Gemini for analysis)
  • Data layer (FAISS vector store, inventory)

While the voice agent handles:

  • Speech and conversation flow
  • Tool orchestration
  • User interaction

Independent human-in-the-loop - The LangGraph interrupt/resume pattern works at the workflow level. The voice agent simply handles the needs_clarification response and calls back with the customer's answer.

┌─────────────────────────────────────────────────────────────┐
│                     Voice Agent (LiveKit)                    │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │ STT/TTS     │  │ Turn        │  │ Tools               │  │
│  │ Pipeline    │  │ Detection   │  │ - inventory_check   │  │
│  │             │  │             │  │ - store_hours       │  │
│  │             │  │             │  │ - project_assistant─┼──┼──┐
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │  │
└─────────────────────────────────────────────────────────────┘  │
                                                                  │
┌─────────────────────────────────────────────────────────────┐  │
│              Project Assistant (LangGraph)                   │◄─┘
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌─────────────┐  │
│  │ Analyze  │─►│ Clarify  │─►│ Identify │─►│ Match       │  │
│  │ Project  │  │ (HITL)   │  │ Materials│  │ Products    │  │
│  └──────────┘  └──────────┘  └──────────┘  └──────┬──────┘  │
│                                                    │         │
│  ┌──────────────────┐  ┌───────────────────────┐  │         │
│  │ Generate         │◄─│ Check Availability    │◄─┘         │
│  │ Shopping List    │  │                       │            │
│  └──────────────────┘  └───────────────────────┘            │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ Data Layer: FAISS Vector Store + Inventory JSON      │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

This pattern scales well - each "assistant" can be developed, tested, and maintained independently, then integrated via a simple tool interface on the main agent.

Store Information Tools

The agent has access to real-time store data through callable tools:

  • Inventory lookup - Check product availability, pricing, and aisle location at specific stores
  • Project assistant - Help customers plan DIY projects with complete shopping lists
  • Store hours - Get operating hours for each location
  • Department info - List available departments (Sales, Tool Rental, Pro Desk, etc.)

Multilingual Support

The agent automatically responds in the caller's language. Using ElevenLabs Scribe v2 for speech-to-text (99+ languages), Google Gemini for understanding and response generation, and Cartesia for text-to-speech, language detection and response happens automatically with no manual selection needed.

Voice AI Pipeline

Coding Agents and MCP

This project is designed to work with Claude Code.

To get the most out of Claude Code, install the LiveKit Docs MCP server:

claude mcp add --transport http livekit-docs https://docs.livekit.io/mcp

The project includes a complete AGENTS.md file. To learn more about this file, see https://agents.md.

Dev Setup

Clone the repository and install dependencies to a virtual environment:

git clone https://github.com/yorrickjansen/livekit-hardware-store-agent.git
cd livekit-hardware-store-agent
uv sync

Sign up for LiveKit Cloud then set up the environment by copying .env.example to .env.local and filling in the required keys:

  • LIVEKIT_URL
  • LIVEKIT_API_KEY
  • LIVEKIT_API_SECRET

You can load the LiveKit environment automatically using the LiveKit CLI:

lk cloud auth
lk app env -w -d .env.local

Project Assistant Setup (Optional)

To enable the project-based shopping assistant, add your Google Gemini API key to .env.local:

GEMINI_API_KEY=your-api-key-here

Get your API key at Google AI Studio.

Then generate the product embeddings (one-time setup):

uv run python scripts/generate_embeddings.py

This creates vector embeddings for ~300 products across 3 store locations, enabling semantic search for the project assistant.

Run the Agent

Before your first run, you must download certain models such as Silero VAD and the LiveKit turn detector:

uv run python src/agent.py download-files

Next, run this command to speak to your agent directly in your terminal:

uv run python src/agent.py console

To run the agent for use with a frontend or telephony, use the dev command:

uv run python src/agent.py dev

In production, use the start command:

uv run python src/agent.py start

Twilio Telephony Setup

To enable phone calls to your agent via Twilio, follow these steps:

Prerequisites

Step 1: Create a Twilio SIP Trunk

twilio api trunking v1 trunks create \
  --friendly-name "My LiveKit Trunk" \
  --domain-name "my-trunk.pstn.twilio.com"

Save the Trunk SID (e.g., TK...) from the output.

Step 2: Configure the Origination URL

Get your LiveKit SIP URI from the LiveKit Cloud dashboard under Telephony → SIP trunks. It looks like sip:abc123.sip.livekit.cloud.

Important: Use the SIP URI shown in the dashboard, not your project subdomain.

twilio api trunking v1 trunks origination-urls create \
  --trunk-sid <your-trunk-sid> \
  --friendly-name "LiveKit SIP URI" \
  --sip-url "sip:<your-sip-uri>.sip.livekit.cloud" \
  --weight 1 --priority 1 --enabled

Step 3: Associate Your Phone Number

# List your phone numbers to get the SID
twilio phone-numbers list

# Associate the phone number with the trunk
twilio api trunking v1 trunks phone-numbers create \
  --trunk-sid <your-trunk-sid> \
  --phone-number-sid <your-phone-number-sid>

Step 4: Create a LiveKit Inbound Trunk

lk sip inbound create --name "Twilio Inbound" --numbers "+1234567890"

Replace +1234567890 with your Twilio phone number.

Step 5: Create a Dispatch Rule

lk sip dispatch create --name "Inbound Calls" --individual "call-" --randomize

Step 6: Deploy Your Agent

lk agent deploy

Verify Setup

After completing these steps, call your Twilio phone number. The call should connect to your LiveKit agent.

To troubleshoot:

  • Check agent status: lk agent status
  • View agent logs: lk agent logs
  • List rooms: lk room list
  • Check LiveKit Cloud Telephony → Calls for call logs

Tests and Evals

This project includes a complete suite of evals, based on the LiveKit Agents testing & evaluation framework. To run them, use pytest.

uv run pytest

License

This project is licensed under the MIT License - see the LICENSE file for details.

About

A voice AI agent for a hardware store, built with LiveKit Agents

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages