Seq2Seq Attention

Attention was born from a simple problem: how do you translate a long sentence without losing information? The answer changed everything about how neural networks process language.

The Encoder-Decoder Architecture

Many tasks involve transforming one sequence into another:

Vanilla Seq2Seq

Encoder:  x₁ → h₁ → h₂ → h₃ → [context vector c]
                                    ↓
Decoder:                      c → s₁ → s₂ → s₃
                                  ↓    ↓    ↓
                                 y₁   y₂   y₃

The encoder processes the entire input and compresses it into a single fixed-size vector $c$. The decoder then generates the output using only this one vector.

The Bottleneck Problem

All information about the entire input must fit in one vector (typically 512 floats). For a 100-word sentence, this means massive information loss — especially for details at the start of the input, which are processed first and most likely to be forgotten.

This is like trying to summarize a novel in a single tweet — something has to give.

Attention: The Solution

Key Insight

Instead of one context vector for the entire output, compute a different context for each decoder step by looking back at all encoder states:

$c_t = \sum_{i=1}^{T_x} \alpha_{t,i} \cdot h_i$

Where $\alpha_{t,i}$ is the attention weight: “how much should decoder step $t$ focus on encoder step $i$?”

Encoder states:     h₁    h₂    h₃    h₄
                     ↓     ↓     ↓     ↓
Attention weights:  0.1   0.6   0.2   0.1  (for decoder step t)
                     ↓     ↓     ↓     ↓
                    ────────────────────
                          context c_t

For translating “world” → “monde”, attention focuses on $h_2$ (where “world” was encoded), not on all encoder states equally.

Interactive: Cross-Attention Alignment

Step through the decoder to see how attention shifts across source words at each generation step:

Attention Mechanisms

Bahdanau (Additive) Attention

The original attention mechanism (Bahdanau et al., 2014) uses a small neural network to compute compatibility:

$e_{t,i} = v^T \tanh(W_s s_{t-1} + W_h h_i)$ $\alpha_{t,i} = \frac{\exp(e_{t,i})}{\sum_j \exp(e_{t,j})}$ $c_t = \sum_i \alpha_{t,i} \cdot h_i$

Luong (Multiplicative) Attention

A simpler variant (Luong et al., 2015) uses dot products instead of a neural network:

The dot product variant became the basis for self-attention in Transformers.

def bahdanau_attention(s_prev, encoder_outputs, Ws, Wh, v):
    """Bahdanau (additive) attention."""
    src_len = encoder_outputs.shape[0]
    scores = np.zeros(src_len)
    for i in range(src_len):
        combined = np.tanh(Ws @ s_prev + Wh @ encoder_outputs[i])
        scores[i] = v @ combined
    weights = softmax(scores)
    context = weights @ encoder_outputs
    return context, weights

def luong_attention(s_curr, encoder_outputs, W=None):
    """Luong (multiplicative) attention."""
    if W is not None:
        scores = encoder_outputs @ W @ s_curr  # General
    else:
        scores = encoder_outputs @ s_curr       # Dot
    weights = softmax(scores)
    context = weights @ encoder_outputs
    return context, weights

Attention Alignments

What Attention Learns

For translation “the cat sat” → “le chat assis”:

Attention learns to align source and target words — without any explicit alignment supervision!

Non-Monotonic Alignment

For “I am hungry” → “J’ai faim” (French has different word order):

Attention can reorder information — “hungry” maps to “faim” which comes in a different position. This is something the fixed context vector in vanilla seq2seq couldn’t do at all.

Key Equations

From Seq2Seq to Self-Attention

Seq2seq attention lets the decoder attend to the encoder. But what if we let every token attend to every other token in the same sequence? That’s self-attention — the core of the Transformer. Next: Self-Attention →