2.1 KiB
2.1 KiB
ARIA — Zero-to-Tiny LLM (Python)
ARIA is a beginner-friendly, step-by-step course that takes you from “Hello World” to training a tiny decoder-only, character-level LLM in Python. Each lesson is a single, runnable file with clear docstrings, doctests where helpful, and minimal dependencies.
Note: This repository’s instructional content was generated with the assistance of an AI language model.
What you’ll build
- A progression of tiny language models:
- Count-based bigram model → NumPy softmax toy → PyTorch bigram NN
- Single-head self-attention → Mini Transformer block
- A tiny decoder-only model trained on a small corpus (e.g., Tiny Shakespeare)
Who this is for
- Beginners who can run
python script.pyand have written a basic “Hello World”. - Learners who want a clear path to an LLM without heavy math or large datasets.
Course outline (lessons)
- Read a Text File (with docstrings)
- Character Frequency Counter
- Train/Val Split
- Char Vocabulary + Encode/Decode
- Uniform Random Text Generator
- Bigram Counts Language Model
- Laplace Smoothing (compare w/ and w/o)
- Temperature & Top-k Sampling
- Perplexity on Validation
- NumPy Softmax + Cross-Entropy (toy)
- PyTorch Tensors 101
- Autograd Mini-Lab (fit y = 2x + 3)
- Char Bigram Neural LM (PyTorch)
- Sampling Function (PyTorch)
- Single-Head Self-Attention (causal mask)
- Mini Transformer Block (pre-LN)
- Tiny Decoder-Only Model (1–2 blocks)
- (Optional) Save/Load & CLI Interface
Each lesson includes: Outcome, Files, Dependencies, Directions, Starter Code with docstrings + doctests, Run, What you learned, Troubleshooting, Mini-exercises, Next lesson.
Requirements
- Python: 3.10+
- OS: Windows/macOS/Linux (UTF-8 locale recommended)
- Dependencies:
- Stdlib only until Lesson 9
- NumPy for Lessons 8–10
- PyTorch (CPU is fine) from Lesson 11 onward
- Hardware: CPU is enough for all lessons; tiny models, short runs
Install common deps (when needed):
pip install numpy torch --upgrade