🎬 vid‑rag — Minimal Multimodal Video Retrieval (Text + Vision)

Engineering‑focused assignment: demonstrate a clear, well‑tested pipeline and a simple UI. Not aiming for SOTA accuracy.

---

TL;DR (Reviewer Quick Guide)

1. Install (Python 3.11+, no system CUDA required):

   uv sync --all-extras --dev

2. Run UI:

   make dev
   # open the Streamlit URL shown in the terminal

3. In UI → “📥 Ingest & Build Index”
   • Pick your video folder (default: data/vids/).
   • Model: tiny or base.
   • (Optional) ✔ Also build vision (CLIP) index if ffmpeg is installed.
   • Click Ingest & Build Index.
4. In UI → “🔎 Search & Watch”
   • Query example: “a brief explanation of Starry Night by Van Gogh”.
   • See top‑K videos with player auto‑seeking to the best segment.
   • (Optional) Toggle vision fusion and adjust vision weight.

If vision prerequisites are missing, the app automatically falls back to text‑only.

---

What This Delivers

• Local ingestion of multiple videos from a folder.
• ASR → Segments (timestamps + text) using Whisper via faster-whisper.
• Text retrieval with MiniLM embeddings.
• Vision (optional): one keyframe per segment, CLIP embeddings; late‑fusion with text.
• Top‑K videos UX: one result per video, pointing to the relevant section (timestamp).
• Streamlit UI with clear controls and feedback.
• Logging & error handling: centralized user‑friendly messages, structured logs.
• Unit tests & coverage: core pipeline covered; fast tests with mocks (no heavy downloads/ffmpeg calls).

---

Project Structure

app/
  main.py                 # Streamlit UI
  errors.py               # AppError types + user-facing mapping (report_error)
  logging_conf.py         # Minimal logger factory (stdout)
  types.py                # Frozen dataclasses (Segment, SearchResult, VideoAsset, ...)

  services/
    ingestion.py          # discover → (probe?) → transcribe → chunk → Segments
    index_build.py        # build_text_index(), build_vision_index()
    embeddings.py         # MiniLM + CLIP wrappers (text / images)
    search.py             # SimpleTextSearcher + LateFusionSearcher (weighted fusion)
    storage.py            # LocalANN (numpy-based ANN index, persist/load/query)
    vision.py             # One-frame extraction via ffmpeg (optional)

  utils/
    io.py                 # path helpers, deterministic IDs
    timing.py             # tiny @timeit decorator
    metrics.py            # perf helper (CPU avg, peak RSS, wall time)

artifacts/                # persisted indices & thumbs (created at runtime)

Artifacts layout (created on first run):

• artifacts/text/ — text ANN: embeddings.npy, ids.json, payloads.json, segments.jsonl
• artifacts/vision/ — vision ANN (if built): same layout
• artifacts/thumbs/ — keyframes {video_id}/{segment_id}.jpg (if vision built)

---

Requirements

• Python 3.11+
• uv (dependency & venv manager): https://docs.astral.sh/uv/
• ffmpeg (optional for vision) — text‑only mode works without it.

Apple Silicon / Linux are both fine. GPU is not required; inference uses CPU by default.

---

Setup & Commands

uv sync --all-extras --dev    # install runtime + dev deps
make dev                      # run UI
make test                     # unit tests (fast; mocks external deps)
make typecheck                # mypy
make lint                     # ruff/isort/black check
make format                   # format code
make coverage                 # coverage report (HTML: htmlcov/)

make bench                    # text-only benchmark (ingest + text index)
make bench-vision             # text + vision benchmark (requires ffmpeg)

---

How It Works (Short)

1. Ingestion
   • Discover videos in a folder.
   • (Optional) Probe basic metadata via ffprobe (duration/FPS) for logging.
   • Transcribe audio → raw pieces (start, end, text).
   • Chunk into Segments with merge/cut heuristics (max_gap_s, max_chars, min_chars).
   • Embed text (MiniLM) and persist a local ANN index.
2. Vision (optional)
   • Extract a single keyframe at each segment’s midpoint.
   • Embed with CLIP; persist a vision ANN index.
3. Search
   • Encode the query (text; and CLIP text if vision enabled).
   • Query both indices; late‑fuse scores: w_text * s_text + w_vision * s_vision.
   • Aggregate to top‑K videos (best segment per video).
   • UI plays the video starting at that segment; lists other matching timestamps.

---

Design Choices (Why This Way)

• Clarity over complexity: local numpy‑based ANN keeps dependencies minimal and code readable.
• Late fusion is simple, tunable, and explainable.
• One keyframe per segment balances cost and signal; vision is optional.
• Top‑K videos matches the spec precisely: one section per returned video.
• Centralized error reporting surfaces clear user messages while logging details for debugging.

---

Performance (warm run)

Machine: Apple M2-series / 16 GB RAM , Python 3.11, device=mps
Corpus: 11 videos, total audio ≈ 641.13 s (~10.7 min), 23 segments

How to reproduce

# text-only
uv run python scripts/bench.py --data data/vids
# text + vision (ffmpeg required)
uv run python scripts/bench.py --data data/vids --vision

Results

• Ingest (ASR+chunking): 30.49 s wall, CPU avg 306%, peak RSS 915 MB – Throughput 0.75 seg/s, xRT 21.03× (audio_seconds / wall_seconds)

• Index (text): 2.83 s wall, CPU avg 13.6%, peak RSS 846 MB – ~123 ms/segment (2827.69 ms / 23)

• Index (vision): 4.30 s wall, CPU avg 16.4%, peak RSS 524 MB – ~187 ms/segment (4297.46 ms / 23)

perf | ingest       | wall_ms=30489.55 cpu_avg_pct=306.59 rss_peak_mb=914.59
perf | index_text   | wall_ms=2827.69  cpu_avg_pct=13.57  rss_peak_mb=846.22
perf | index_vision | wall_ms=4297.46  cpu_avg_pct=16.42  rss_peak_mb=524.09

Wall time ~linear with total audio duration (ingest) and #segments (indexing). CPU% >100 = multi-core. “Warm” run excludes one-time model downloads.

---

Notes for Reviewers

• No sample media provided in the repo; point the UI to any local folder containing common formats (.mp4, .mov, .mkv, .webm).
• If ffmpeg is missing, vision is skipped; text‑only still demonstrates the pipeline end‑to‑end.
• Tests are hermetic: models/ffmpeg are mocked; runs in seconds.
• Coverage gate (if CI enabled) is set to a reasonable threshold (≥85%) on the core modules.

---

Troubleshooting

• No segments created → ensure the folder path is correct and videos are supported; check logs.
• Vision not active → install ffmpeg or uncheck the vision option.
• Playback warning → UI resolves the file path from segment payloads; don’t move files after ingest.

---

Limitations & Future Work

• Silent videos aren’t segmented by ASR; a simple enhancement would create vision‑only synthetic segments.
• Scaling: LocalANN is fine for small/mid corpora; swapping to FAISS/Qdrant is straightforward.
• Keyframe downscaling would speed up vision on 4K sources (kept out for simplicity; easy to add in vision.py).