LLM Notebooks

---

Overview

LLM Notebooks is a structured collection of Jupyter notebooks documenting a ground-up study of transformer architectures, large language models, fine-tuning techniques, and the application patterns built on top of them — from basic NLP, through parameter-efficient fine-tuning (PEFT), to retrieval-augmented generation (RAG) and multi-agent systems.

Topics covered include:

• Building sequence models from scratch in PyTorch
• Implementing transformer encoder and decoder architectures
• Pretraining and fine-tuning BERT and GPT-style models
• Using the HuggingFace ecosystem for inference and supervised fine-tuning
• Adapter- and LoRA-based PEFT for parameter-efficient task adaptation
• Building retrieval-augmented generation (RAG) pipelines from scratch
• Designing simple multi-agent systems with free, keyless tools

---

Features

• Built from scratch — core architectures implemented in PyTorch without shortcuts
• Progressive curriculum — each notebook builds on the previous one
• Documented — inline comments explain every design decision
• Colab-ready — every notebook runs on Google Colab free tier (CPU or T4 GPU)

---

Notebook Curriculum

01 — Foundations

#	Notebook	Key concepts
01	Text classification	Bag of words, TF-IDF, logistic regression
02	Feedforward language model	N-gram prediction, cross-entropy loss
03	Word2Vec and embeddings	Semantic vector spaces, GloVe

02 — Sequence Models

#	Notebook	Key concepts
04	Seq2Seq translation	GRU encoder-decoder, teacher forcing, hidden state bottleneck

03 — Transformers

#	Notebook	Key concepts
06	Transformer text classification	TransformerEncoder, positional encoding, GloVe embeddings
07	GPT decoder language model	Causal masking, autoregressive generation
08	BERT pretraining	Masked language modeling, bidirectional attention
09	BERT data preparation	Tokenization pipelines, attention masks
10	Transformer translation	Cross-attention, BLEU score, greedy decoding

04 — Fine-tuning

#	Notebook	Key concepts
11	HuggingFace inference	Pipeline API vs manual inference, DistilBERT, GPT-2
12	Fine-tuning with PyTorch	Transfer learning, requires_grad, AG News → SST-2
13	Fine-tuning with HuggingFace	SFTTrainer, AdamW, BERT emotion classification
14	Adapters and PEFT	Bottleneck adapters, FeatureAdapter, 0.1% trainable parameters
15	LoRA from scratch	Low-rank adaptation, q/v projection injection, 1.1% trainable parameters

05 — RAG and Agents

#	Notebook	Key concepts
16	AI agents from scratch	Multi-agent design, TF-IDF retrieval, free keyless tool integration
17	RAG with DPR and FAISS	Dense passage retrieval, FAISS index, dot product vs cosine similarity
18	RAG from scratch (BERT)	Mean-pooled BERT embeddings, attention masking, manual retrieval pipeline

---

Installation

git clone https://github.com/mminotaki/llm-notebooks
cd llm-notebooks

---

Usage

jupyter notebook 03_transformers/10_transformer_translation.ipynb

Each notebook installs its own dependencies in the first cell. No global installation required.

---

Dependencies

Package	Purpose
torch	Core deep learning framework
torchtext	Text tokenization and GloVe embeddings
transformers	HuggingFace pretrained models
datasets	HuggingFace dataset hub
trl	Supervised fine-tuning trainer (SFTTrainer)
torchmetrics	Evaluation metrics
scikit-learn	TF-IDF and similarity utilities (notebook 16)
faiss-cpu	Vector index for dense retrieval (notebook 17)
requests	Free keyless API calls in the agents notebook (notebook 16)

---

License

This project is licensed under the MIT License — see the LICENSE file for details.