Positional Encoding

The Problem: Attention Doesn’t Know Order

Here’s a fundamental issue with self-attention: it’s permutation-invariant.

Consider these two very different sentences:

“Dog bites man” → Routine news
”Man bites dog” → Breaking news!

If you compute self-attention on both, the attention weights between “dog”, “bites”, and “man” are identical — because attention only cares about what the tokens are, not where they are.

Self-attention sees the same set of tokens regardless of order — clearly wrong. Word order matters enormously. We need to inject position information.

Why RNNs Don’t Have This Problem

RNNs process tokens sequentially — position is implicit in the computation order. Token 3 is processed after token 2, and the hidden state carries temporal information.

Transformers process all positions in parallel — fast, but position-blind without help.

The Solution: Add Position Vectors

The idea is simple: create a positional encoding vector for each position, and add it to the token embedding:

$x_{\text{input}} = \text{token\_embedding} + \text{positional\_encoding}$

Now “dog” at position 0 has a different representation than “dog” at position 2, even though they’re the same word.

Sinusoidal Positional Encoding

The original Transformer paper (2017) used a clever encoding based on sine and cosine waves at different frequencies:

$PE_{(\text{pos}, 2i)} = \sin\left(\frac{\text{pos}}{10000^{2i/d}}\right)$ $PE_{(\text{pos}, 2i+1)} = \cos\left(\frac{\text{pos}}{10000^{2i/d}}\right)$

The Clock Analogy

Think of positional encoding like the hands on a clock:

Together, all the hands uniquely identify any moment in time. Similarly, the combination of all sine/cosine waves at different frequencies uniquely identifies each position.

def sinusoidal_positional_encoding(max_len, d_model):
    PE = np.zeros((max_len, d_model))
    position = np.arange(max_len)[:, np.newaxis]
    
    # Different frequency for each dimension pair
    div_term = np.exp(np.arange(0, d_model, 2) * (-np.log(10000.0) / d_model))
    
    PE[:, 0::2] = np.sin(position * div_term)  # Even dims: sin
    PE[:, 1::2] = np.cos(position * div_term)  # Odd dims: cos
    
    return PE

Visualize the Encoding

Explore the sinusoidal positional encoding patterns. Switch between heatmap view (see all dimensions at once) and wave view (see how individual dimensions vary with position):

What to notice:

• Low dimensions (left columns) change rapidly — high-frequency “second hand”
• High dimensions (right columns) change slowly — low-frequency “hour hand”
• Each position has a unique combination of values across all dimensions

Why Do Nearby Positions Look Similar?

The key property of sinusoidal encoding: similarity between positions depends on their distance, not their absolute values.

Hover over the matrix below to see how cosine similarity between position vectors decreases smoothly with distance:

Why This Is Useful

This relative-distance property means:

• The model can learn that “the word 2 positions to my left is usually a verb”
• This pattern works the same at position 5 as at position 500
• The model doesn’t have to relearn positional patterns for every absolute position

Comparison: Different Position Encodings

Learned Positional Embeddings

BERT and GPT-2 simply learn a $(max\_len, d_{model})$ parameter matrix — each position gets a trainable vector.

Pros: Slightly better performance (model optimizes the encoding)
Cons: Can’t handle sequences longer than max_len

class LearnedPE:
    def __init__(self, max_len, d_model):
        self.PE = np.random.randn(max_len, d_model) * 0.01  # Learned!
    
    def forward(self, seq_len):
        return self.PE[:seq_len]

RoPE (Rotary Position Embedding)

The modern favorite. Instead of adding PE to the embedding, RoPE rotates the Q and K vectors based on their position:

$q'_m = q_m \cdot e^{im\theta}, \quad k'_n = k_n \cdot e^{in\theta}$

The dot product then naturally encodes relative position:

$q'_m \cdot k'_n = q_m \cdot k_n \cdot e^{i(m-n)\theta}$

The phase difference $(m-n)$ captures how far apart the tokens are. This is elegant because relative position information emerges automatically in the attention scores.

Used by: LLaMA, LLaMA 2, Mistral, Gemma, and most modern LLMs.

Key Equations

Sinusoidal

$PE_{(\text{pos}, 2i)} = \sin\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$ $PE_{(\text{pos}, 2i+1)} = \cos\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$

Input to Transformer

$x = \text{Embedding}(\text{token}) + \text{PE}(\text{position})$

Summary

1. Self-attention is position-blind — we must add position information explicitly
2. Sinusoidal encoding uses different-frequency waves to create unique position signatures
3. Nearby positions are similar — the encoding captures relative distance
4. Modern models use RoPE — it encodes relative position directly in attention scores

Next: Layer Normalization & Residual Connections

The final building blocks for stable training: Layer Norm & Residuals →