Positional Encoding

What it is

Positional encoding is how transformers understand word order. Since transformers process all tokens simultaneously (unlike sequential models), they need another way to know that "dog bites man" differs from "man bites dog." Positional encodings add information about each token's position in the sequence.

Why it matters

Positional encoding enables transformers to understand sequence and structure. Without it, the model would see sentences as bags of words with no order. Modern variations allow for very long context windows by using efficient positional encoding schemes like RoPE (Rotary Position Embedding).