Architecture Overview

You’ve now learned every individual component of a transformer:

There’s one piece we haven’t covered yet — the Feed-Forward Network — and then we’ll assemble everything.

The Missing Piece: Feed-Forward Network (FFN)

After self-attention lets tokens gather information from each other, each token needs to process that information independently. That’s the FFN’s job.

What It Does

The FFN applies two linear transformations with a non-linearity in between:

$\text{FFN}(x) = \text{ReLU}(x W_1 + b_1) W_2 + b_2$

Think of it this way:

• Attention = “gather information from other tokens” (communication)
• FFN = “think about what you’ve gathered” (computation)

The Bottleneck Architecture

The FFN expands the dimension, then compresses it back:

Why expand then contract? The wider hidden layer lets the network learn more complex transformations. It’s like brainstorming widely then filtering down to the best ideas.

A Surprising Fact

The FFN often has more parameters than the attention layers!

The FFN uses roughly twice the parameters of attention in each block.

class FeedForward:
    def __init__(self, d_model, d_ff):
        self.W1 = np.random.randn(d_model, d_ff) * 0.01
        self.b1 = np.zeros(d_ff)
        self.W2 = np.random.randn(d_ff, d_model) * 0.01
        self.b2 = np.zeros(d_model)
    
    def forward(self, x):
        # Expand → ReLU → Contract
        hidden = np.maximum(0, x @ self.W1 + self.b1)  # ReLU
        return hidden @ self.W2 + self.b2

Anatomy of a Transformer Block

Every transformer block follows the same pattern — two sub-layers, each wrapped with a residual connection and layer normalization:

$x' = x + \text{MultiHeadAttn}(\text{LN}(x))$ $\text{output} = x' + \text{FFN}(\text{LN}(x'))$

This is the Pre-LN variant (used by GPT-2, GPT-3, LLaMA). The original 2017 paper used Post-LN (normalize after the residual add), but Pre-LN trains more stably.

class TransformerBlock:
    def __init__(self, d_model, n_heads, d_ff):
        self.attn = MultiHeadAttention(d_model, n_heads)
        self.ffn = FeedForward(d_model, d_ff)
        self.ln1 = LayerNorm(d_model)
        self.ln2 = LayerNorm(d_model)
    
    def forward(self, x, mask=None):
        # Sub-layer 1: Attention
        x = x + self.attn(self.ln1(x), mask)
        
        # Sub-layer 2: FFN
        x = x + self.ffn(self.ln2(x))
        
        return x

The Three Transformer Variants

Click through the variants below to see how the same building blocks assemble differently:

Encoder-Decoder (Original, T5, BART)

The original “Attention Is All You Need” design:

• Encoder: Processes the full input with bidirectional self-attention
• Decoder: Generates output one token at a time with causal self-attention + cross-attention to encoder
• Best for: Translation, summarization — tasks where input and output are different sequences

Encoder-Only (BERT)

Just the encoder stack:

• Bidirectional attention: Every token sees every other token
• Can’t generate text directly (no autoregressive decoding)
• Best for: Classification, named entity recognition, sentence similarity
• Key insight: Understanding a sentence benefits from seeing the full context

Decoder-Only (GPT, LLaMA, Claude)

Just the decoder stack (no cross-attention):

• Causal attention: Each token can only see previous tokens
• Generates text autoregressively — one token at a time
• Best for: Text generation, chat, code completion — and it turns out, almost everything
• Why it won: Simpler architecture + scales better + can do few-shot learning

The Evolution of Transformers

The trend is clear: decoder-only architectures won. Nearly all modern LLMs use this simpler design.

Next: End-to-End Walkthrough

Now that you understand the architecture, let’s trace data through a complete transformer — from input text to predicted next token. →