Build a speech-to-text LLM from scratch, and train it on one GPU.
tiny-audio is to speech what nanoGPT is to text: a small, readable, hackable speech LLM (a pretrained audio encoder, a tiny MLP projector, and an open-source LLM decoder) plus a course that teaches you to build, train, evaluate, and ship it yourself. The model trained with this repo's default recipe gets 1.8% WER on LibriSpeech test-clean while training only ~80M parameters.
It is deliberately simple. For production speech recognition, use a mature toolkit or a hosted API. This repo is for understanding how the pieces fit, and for running cheap experiments of your own.
tiny-audio is a personal side project, independent of my work at AssemblyAI.
Course: alexkroman.github.io/tiny-audio · Live demo: huggingface.co/spaces/mazesmazes/tiny-audio · Model: mazesmazes/tiny-audio
# 1. Transcribe with the published model (the first call downloads ~6 GB)
# pip install "transformers>=5.0" peft torch torchaudio librosa
from transformers import pipeline
pipe = pipeline(
"automatic-speech-recognition", model="mazesmazes/tiny-audio", trust_remote_code=True
)
print(pipe("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")["text"])
# I have a dream that one day this nation will rise up and live out the true meaning of its creed.Pass a path to transcribe your own recording. WAV, FLAC, OGG and MP3 work out of the box; other formats (m4a, video) need ffmpeg installed.
# 2. Train your own for free, on your laptop: ~30 minutes on an Apple Silicon Mac, ~3 GB download
# Needs Python 3.12 and Poetry (pip install poetry)
git clone https://github.com/alexkroman/tiny-audio.git && cd tiny-audio
poetry install
poetry run ta train asr -e tiny
poetry run ta eval -m outputs/tiny --local-code -d librispeech -n 20 # read its transcripts
# 3. The full run on one rented GPU
poetry run ta runpod plan # VRAM and disk the config needs, before you rent anything
poetry run python scripts/train.py # the default config: the published model's recipeThe free run (configs/experiments/tiny.yaml) is the default recipe with the smaller Qwen3.5-0.8B decoder, trained on LibriSpeech dev-clean (5.4 hours) instead of the full mix. Watch the loss: it sits near 4 while the projector's output is still noise to the decoder, then drops below 1 around step 500 as the decoder starts reading the audio. The result transcribes unseen LibriSpeech audio at about 16% WER, with mistakes like "furnishings" for "furniture". It fits in 16 GB of memory and also runs on a CUDA GPU, including a free Colab T4.
What runs where:
| Step | Apple Silicon (16 GB) | Any CUDA GPU (incl. Colab T4) | CPU only | One 80 GB GPU |
|---|---|---|---|---|
| 1. Transcribe | Yes | Yes | Yes, slow | Yes |
2. Free run (-e tiny) |
Yes, ~30 min | Yes | Slow | Yes |
| 3. Full run (default config) | No | Needs 80 GB | No | Yes, a day+ |
No install at all: the live demo lets you record yourself or upload a file.
The idea fits in one class. tiny_audio/projectors.py holds the projector, the only part trained from scratch: frame stacking plus a two-layer MLP, about twenty lines of code under a docstring that records what was tried and why it was rejected. Read it first.
Audio (16 kHz) → speech encoder (frozen) → MLP projector (trained) → LLM decoder (LoRA) → Text
- A pretrained speech encoder turns audio into a sequence of frame embeddings.
- The projector stacks neighbouring frames and maps them into the LLM's embedding space.
- The LLM reads those projected frames as if they were words and writes out the transcript.
Then follow one clip through tiny_audio/asr_modeling.py:
_encode_audio runs the encoder and the projector, forward splices the audio embeddings into the
prompt and computes the loss, and generate decodes. That file is the model that was trained and
published, so it is longer: it also handles batching, LoRA, memory, and several encoder and decoder
families.
Each chapter is a section of the course with code you run and something concrete to check.
| # | Chapter | What you build or learn | Check |
|---|---|---|---|
| 00 | What a speech LLM is | Encoder → projector → decoder, and why reusing two models works | — |
| 01 | Audio in, features out | Waveform → spectrogram → encoder frames, with real shapes | Frame counts per second of audio |
| 02 | The projector | Frame stacking and the two-layer MLP, with real shapes | Shapes and parameter count |
| 03 | The decoder | How an LLM reads audio tokens next to text tokens | — |
| 04 | Run inference | Install, transcribe, timestamps, explore the CLI | Your first transcript |
| 05 | Data | Why the label conventions decide the result | — |
| 06 | Training on one GPU | Size a GPU, write a config, launch on RunPod, read the curves | Loss and gradient curves in W&B |
| 07 | Evaluating | WER, normalization, error analysis, commercial baselines | Your WER next to the table below |
| 08 | Shipping it | Publish to the Hub, deploy a demo, run the HTTP server | A live demo of your own model |
Coming next: per-speaker transcription (diarization), what changed between releases and why, and where to go after this (bigger encoders, streaming, post-training).
Word error rate (%, lower is better) of the published model, trained with the default config, on
11,822 samples (up to 1,000 per dataset), measured with this repo's ta eval against the ta serve
HTTP API on an RTX 4090. A course run trains on far less data, so expect a higher WER from yours.
| Dataset | WER |
|---|---|
| LibriSpeech test-clean | 1.84 |
| SPGISpeech | 2.29 |
| TED-LIUM | 3.71 |
| LoquaciousSet † | 6.20 |
| LibriSpeech test-other | 6.38 |
| VoxPopuli | 7.11 |
| Common Voice | 7.18 |
| AMI (IHM) | 8.99 |
| GigaSpeech | 9.06 |
| Earnings22 † | 10.58 |
| People's Speech | 17.59 |
| AMI (SDM) | 23.53 |
| Mean (12 sets) | 8.71 |
| Pooled (11,822 samples) | 7.42 |
† Held out: no data from this source was used in training.
Check these numbers yourself, and compare against commercial APIs on the same samples:
poetry run ta eval -m mazesmazes/tiny-audio -d loquacious -n 100
# Same samples through a commercial API (also: deepgram, elevenlabs, apple-speech)
ASSEMBLYAI_API_KEY=... poetry run ta eval -m assemblyai -d loquacious -n 100What each training budget costs:
| Run | Data | Hardware | Cost |
|---|---|---|---|
Free run (-e tiny) |
LibriSpeech dev-clean (5.4 hours) | Your laptop or a free Colab GPU | Free |
| Course run | LoquaciousSet small (~250 hours) |
One rented GPU, a few hours | A few GPU-hours |
| Default config (published model) | ~3M clips across ten corpora (>1 TB) | One 80 GB GPU, a day or more | Hundreds of dollars |
# Word-level timestamps (forced alignment)
pipe("audio.wav", return_timestamps=True)
# {"text": "hello world", "words": [{"word": "hello", "start": 0.0, "end": 0.5}, ...]}
# Who spoke when (speaker diarization)
pipe("meeting.wav", return_speakers=True, num_speakers=2)Transcripts come punctuated, capitalized, and with numbers formatted. The input can be a file path,
a URL, or a 16 kHz numpy array; the bf16 weights need roughly 6 GB of GPU or Apple Silicon memory.
Speaker diarization needs transformers installed from main
(pip install git+https://github.com/huggingface/transformers) until the next release. For
token-by-token output, see ASRModel.generate_streaming.
- HTTP server:
ta servebatches concurrent requests on one GPU, locally or on RunPod. See docs/SERVING.md. - Batching and GPU settings: the model card.
tiny_audio/ the model: projector, encoder/decoder glue, pipeline, diarization
scripts/train.py the training loop (Hydra + HF Trainer)
configs/ the default recipe, its data mix, and the free `tiny` preset
scripts/ the `ta` CLI: eval, serve, push, deploy, runpod
docs/course/ the course
demo/ the Hugging Face Space
Every training setting is a Hydra override, for example
model.projector_hidden_dim=2048. ta push publishes a model to the Hub and ta deploy puts a
demo like the one above on a Space.
| Problem | Fix |
|---|---|
poetry install refuses the Python version |
Install Python 3.12, then poetry env use python3.12 |
| Model download fails | poetry run hf auth login, or export HF_TOKEN |
flash_attention_2 warning |
Expected off CUDA; the model falls back to SDPA |
| Out of GPU memory during the free run | Halve the batch and keep the effective batch of 4: |
poetry run python scripts/train.py +experiments=tiny \
training.per_device_train_batch_size=1 training.gradient_accumulation_steps=4Anything else: check Common issues, then open an issue with the command you ran and its output.
Want to try a new projector, add a dataset, or change the codebase? See CONTRIBUTING.md for the CLI reference, config layout, and quality gates.
- Granite Speech and GLM-ASR for audio encoding
- Qwen3.5 and Qwen3 for language modeling
- LoquaciousSet for the default evaluation set
- nanoGPT, whose small-and-readable approach this repo borrows
MIT