Benchmarks

What it is

Benchmarks are standardised tests that measure AI model capabilities across specific tasks—reasoning, coding, maths, language understanding, etc. They provide comparable scores so you can evaluate how models stack up. Examples include MMLU, HumanEval, and GSM8K.

Why it matters

Benchmarks help you choose models based on capabilities that matter for your use case. However, they have limitations—models can be optimised for benchmarks specifically, and scores don't always reflect real-world performance. Use benchmarks as one data point, not the final word.