llm-demo

A full-stack local LLM chat application — ChatGPT-style UI powered by your own machine.

---

Architecture

┌─────────────────────────────────────┐
│  customGPT  (React Native · :8081)  │
│                                     │
│  Header · ChatInput · MessageBubble │
│  Streaming cursor · Theme toggle    │
└────────────────┬────────────────────┘
                 │
                 │  POST /chat/completions
                 │  stream: true  →  NDJSON chunks
                 │
┌────────────────▼────────────────────┐
│  llm-api  (FastAPI · :8000)         │
│                                     │
│  CORS · Logging · /chat/completions │
│  ollama_client  (httpx stream)      │
└────────────────┬────────────────────┘
                 │
                 │  POST /api/generate
                 │  stream: true  →  NDJSON chunks
                 │
┌────────────────▼────────────────────┐
│  Ollama  (LLM Server · :11434)      │
│                                     │
│  llama3.2 · codellama · mistral     │
│  Apple Silicon GPU (Metal)          │
└─────────────────────────────────────┘

Data Flow

User types prompt
  │
  ▼
customGPT — useChat → streamMessage()
  │
  │  POST /chat/completions { prompt, model, stream: true }
  ▼
llm-api — logs request → stream_generate()
  │
  │  POST /api/generate { prompt, model, stream: true }
  ▼
Ollama — runs inference on GPU
  │
  │  { "response": "Hello",  "done": false }
  │  { "response": " world", "done": false }
  │  { "response": "",       "done": true  }
  ▼
llm-api — StreamingResponse (application/x-ndjson)
  ▼
customGPT — appends each token in real time
  ▼
User sees text stream with ▌ cursor + syntax-highlighted code

---

Quickstart

One command does everything — installs missing tools, pulls the model, and starts all services.

git clone <repo-url>
cd llm-demo
chmod +x gpt.sh
./gpt.sh start

The browser opens automatically at http://localhost:8081 when everything is ready.

---

gpt.sh — Management Script

gpt.sh is the single entry point for running the entire stack.

Usage:  ./gpt.sh <command>

  start   Install dependencies, pull LLM model, launch all services
  stop    Stop all running services (Ollama, llm-api, customGPT)

./gpt.sh start

On first run (or after a fresh clone) the script automatically:

Step	What it does
OS check	Exits early if not macOS
Homebrew	Installs if missing
Node.js ≥ 18	Installs or upgrades via Homebrew
Python ≥ 3.11	Installs or upgrades via Homebrew
Poetry	Installs via Homebrew
Ollama	Installs via Homebrew
poetry install	Installs all Python deps for llm-api
.env setup	Copies llm-api/.env.example → .env if missing
npm install	Installs all Node deps for customGPT
Ollama server	Starts server, waits until healthy
Model pull	Downloads llama3.2 (~2 GB, first run only)
Port check	Warns if 8000 or 8081 are already occupied
Services	Starts Ollama, llm-api, and customGPT in the background
Browser	Opens http://localhost:8081 automatically

On subsequent runs all tools and the model are already present — it goes straight to starting the services.

./gpt.sh stop

Gracefully stops all services:

• Kills the process on port 8000 (llm-api / uvicorn)
• Kills the process on port 8081 (Expo web)
• Kills Metro bundler ports 19000–19006 if open
• Sends SIGTERM → SIGKILL to the ollama process
• Clears all log files for a clean next start

---

Logs

All services write logs to the logs/ folder in the repo root (git-ignored).

File	Service
logs/ollama.log	Ollama LLM server
logs/api.log	llm-api FastAPI server
logs/web.log	customGPT Expo web bundler
logs/services.pid	PIDs of background processes

Tail logs in real time:

tail -f logs/api.log
tail -f logs/web.log
tail -f logs/ollama.log

---

Services

Service	URL	Description
CustomGPT UI	http://localhost:8081	React Native Web chat interface
llm-api	http://localhost:8000	FastAPI backend
API Docs	http://localhost:8000/docs	Swagger UI
Ollama	http://localhost:11434	Local LLM server

---

Prerequisites (auto-installed by gpt.sh start)

• macOS 11 (Big Sur) or later
• Apple Silicon (M1/M2/M3) recommended — Intel Macs supported
• At least 8 GB RAM (16 GB+ recommended)
• Internet connection on first run (for Homebrew + model download)

---

Project Structure

llm-demo/
├── gpt.sh              ← start / stop script
├── logs/               ← runtime logs (git-ignored)
│   ├── ollama.log
│   ├── api.log
│   └── web.log
├── llm-api/            ← FastAPI Python backend
│   ├── app/
│   │   ├── main.py
│   │   ├── config.py
│   │   ├── models.py
│   │   ├── ollama_client.py
│   │   ├── middleware/
│   │   │   └── logging_middleware.py
│   │   └── routers/
│   │       └── chat.py
│   ├── pyproject.toml
│   └── .env.example
└── customGPT/          ← React Native Web UI
    ├── App.tsx
    ├── src/
    │   ├── screens/    ← ChatScreen
    │   ├── components/ ← MessageBubble, ChatInput, Header, …
    │   ├── hooks/      ← useChat (streaming)
    │   ├── services/   ← api.ts (fetch + SSE stream)
    │   ├── context/    ← ThemeContext (light / dark)
    │   └── constants/  ← theme colours, spacing
    └── package.json

---

Manual Setup (without gpt.sh)

If you prefer to run services manually:

# 1. Install Ollama and pull a model
brew install ollama
ollama serve &
ollama pull llama3.2

# 2. Start llm-api
cd llm-api
poetry install --no-root
cp .env.example .env
poetry run uvicorn app.main:app --reload --port 8000

# 3. Start customGPT (new terminal)
cd customGPT
npm install
npx expo start --web

---

Available LLM Models

Model	Pull command	Best for
llama3.2	ollama pull llama3.2	General purpose (default)
llama3.2:3b	ollama pull llama3.2:3b	Low RAM machines
codellama	ollama pull codellama	Code generation
mistral	ollama pull mistral	Fast, versatile
gemma3	ollama pull gemma3	Google's Gemma

Switch models at runtime using the model picker in the UI header.

Browse all models at ollama.com/library.