Loss Function

What it is

A loss function measures how wrong a model's predictions are. During training, the model tries to minimise this loss. For language models, the loss typically measures how surprised the model was by the actual next token compared to its predictions. Lower loss means better predictions.

Why it matters

Loss is the primary metric during training—it tells you if the model is learning. When people discuss training progress, they often reference loss curves. A decreasing loss means the model is improving. Understanding loss helps you interpret training reports and understand what "better" means for AI models.