Linear Algebra for Transformers

Develop a feel for what happens inside a transformer.

📓 Companion notebook: EXDai/linear-algebra-transformers — download and run every code snippet yourself. All plots in this article are generated by the notebook.

Episode 11 of the EXD

---

1. Vectors Are Just Lists of Numbers

A token embedding is a vector — a list of 2048 numbers. But we'll start with two dimensions so we can draw it.

cat is the arrow [0.8, 0.3]. dog is [0.6, 0.7]. Each is just two numbers.

In a real transformer, every token in the sequence gets its own 2048-dimensional vector. The math we're about to do — rotation, dot products, matrix multiplication — generalizes directly to 2048D. Only the shapes of the matrices matter.

---

2. Rotation (Ep10 Recap)

Rotation is a matrix multiply where the matrix is orthogonal — it preserves a vector's length. In a transformer, RoPE rotates Q and K by position-dependent angles so attention knows token order.

Apply it to cat:

The vector traces a circle. Its length (|cat|) stays exactly the same at every angle — rotation is an isometry, moving the vector without stretching or shrinking. This is why values stay bounded even after being rotated across dozens of transformer layers.

In Qwen3.6, only 64 of the 256 dimensions per head are rotated (25% partial rotary factor). The other 192 are position-free — they carry pure semantic content. See Ep10 for the full deep dive into mRoPE.

---

3. The Dot Product: How Similar Are Two Vectors?

The dot product multiplies corresponding elements and sums them:

$$\mathbf{a}\cdot \mathbf{b}=a_1{b}_1+a_2{b}_2+\cdots +a_n{b}_n=\sum _i{a}_i{b}_i\backslash mathbf\{a\}\; \backslash cdot\; \backslash mathbf\{b\}\; =\; a\_1\; b\_1\; +\; a\_2\; b\_2\; +\; \backslash cdots\; +\; a\_n\; b\_n\; =\; \backslash sum\_i\; a\_i\; b\_i$$

Geometrically: a · b = |a| |b| cos(θ). It measures how much two vectors point in the same direction.

For our toy vectors:

cat · dog        = 0.8×0.6 + 0.3×0.7 = 0.69
cat · cat        = 0.8² + 0.3² = 0.73   (= |cat|²)
dog · dog        = 0.6² + 0.7² = 0.85   (= |dog|²)
cat · (−cat)     = −0.73                 (opposite direction)

In a transformer, the attention score between two tokens is a dot product: scoreᵢⱼ = Qᵢ · Kⱼ. Every pair of tokens gets a score. Here's the full pairwise grid for our toy vectors:

This grid is the QKᵀ attention score matrix — just with 2D toy vectors instead of 256D head vectors. Every cell is a dot product. High scores (red) mean the model considers those tokens related. Softmax converts these to probabilities.

But there's a problem. Look at the diagonal: cat·cat = 0.73 while dog·dog = 0.85. Same operation (self-similarity), different results. Something is wrong.

---

4. The Problem: Length Distorts Similarity

a·a = |a|². So dog·dog is larger simply because dog is a longer vector (|dog| = 0.92 vs |cat| = 0.85).

In a transformer, this means a token with a large-magnitude key vector would dominate the softmax — not because it's semantically relevant, but purely because its vector is longer. That would make attention unstable.

The fix: normalize vectors to unit length. Then a·b depends only on the angle between them, not their lengths.

Divide each vector by its length:

cat_norm   = cat / |cat|   = [0.936, 0.351]
dog_norm   = dog / |dog|   = [0.651, 0.760]

Now every self-dot is exactly 1.0, and the pairwise scores measure pure directional similarity:

Every cell on the diagonal is 1.0. cat_norm · dog_norm = 0.876 — this is the cosine of the angle between them (θ = 28.9°). The comparison now depends purely on semantic direction, not vector magnitude.

This is exactly why Qwen3.5 applies QK Norm — RMSNorm on Q and K before computing attention scores. Each head's vectors are scaled to unit variance so the dot product measures direction, not magnitude. V is deliberately not normalized — the model can still control how much information a token carries through the magnitude of V.

$$\text{RMSNorm}(x)=\frac{x}{\sqrt{\text{mean}(x^2)+ϵ}}\cdot \gamma \backslash text\{RMSNorm\}(x)\; =\; \backslash frac\{x\}\{\backslash sqrt\{\backslash text\{mean\}(x^2)\; +\; \backslash epsilon\}\}\; \backslash cdot\; \backslash gamma$$

The learned parameter γ lets the model rescale after normalization.

---

5. Matrix × Vector: Transforming One Vector Into Another

A matrix is a rectangular grid of learned numbers. Multiply it by a vector to get a new vector — possibly with a different number of dimensions.

$$\mathbf{y}=\mathbf{W}\mathbf{x}\text{where}{y}_i=\sum _j{W}_{ij}{x}_j\backslash mathbf\{y\}\; =\; \backslash mathbf\{W\}\backslash mathbf\{x\}\; \backslash quad\backslash text\{where\}\backslash quad\; y\_i\; =\; \backslash sum\_j\; W\_\{ij\}\backslash ,\; x\_j$$

Here's a concrete example: a 2×3 matrix transforms a 3D vector into a 2D vector:

Each output element is a weighted sum of all input elements. The weights (matrix entries) are learned during training.

The shape rule: if W is (m × n) and x is (n,), the result is (m,). The inner dimensions must match. This determines every projection shape in Qwen3.6:

In a transformer, every projection (W_Q, W_K, W_V, W_O) is a matrix×vector multiply. The hidden state h (2048D) is multiplied by learned weight matrices to produce Q (4096D + 4096D gate), K (512D), V (512D), and the output (2048D).

---

6. Projecting Up: Going to a Higher Dimension

When W has more rows than columns (a tall matrix), the output has more dimensions than the input. The vector is "projected up."

We'll project our 2D cat into 3D so we can see it happen:

The orange vector has been lifted off the floor into a third dimension. The original 2D information is still there — you can see its shadow on the xy-plane — but now there's a z-component adding expressive capacity.

This is what the Q projection does: it gives each token more room by projecting into a higher-dimensional space (2048D → 8192D). That extra space is where 16 attention heads can work simultaneously, each operating on a 256-dimensional slice of the projected vector.

---

7. Projecting Down: Squeezing Back to Fewer Dimensions

When W has fewer rows than columns (a wide matrix), the output has fewer dimensions. The vector is "projected down" — compressed.

We project our 3D vector back to 2D. Recall from Section 6:

$$\text{v\_3d}=\left[\begin{array}{c}0.51\\ 0.29\\ 0.55\end{array}\right]\backslash text\{v\backslash \_3d\}\; =\; \backslash begin\{bmatrix\}\; 0.51\; \backslash \backslash \; 0.29\; \backslash \backslash \; 0.55\; \backslash end\{bmatrix\}$$

Now apply W_down (2×3) to get a 2D output:

$$\text{W\_down}\cdot \text{v\_3d}=\left[\begin{array}{c}0.4(0.51)+0.5(0.29)+0.2(0.55)\\ 0.3(0.51)+0.1(0.29)+0.6(0.55)\end{array}\right]=\left[\begin{array}{c}0.459\\ 0.512\end{array}\right]\backslash text\{W\backslash \_down\}\; \backslash cdot\; \backslash text\{v\backslash \_3d\}\; =\; \backslash begin\{bmatrix\}\; 0.4(0.51)\; +\; 0.5(0.29)\; +\; 0.2(0.55)\; \backslash \backslash \; 0.3(0.51)\; +\; 0.1(0.29)\; +\; 0.6(0.55)\; \backslash end\{bmatrix\}\; =\; \backslash begin\{bmatrix\}\; 0.459\; \backslash \backslash \; 0.512\; \backslash end\{bmatrix\}$$

The round-trip output [0.459, 0.512] is visibly different from the original [0.8, 0.3]. Information was lost in compression — the 3D vector held more than could fit back into 2D.

This is what the O projection does: it takes the concatenated head outputs (4096D) and compresses them back into the residual stream (2048D). Sixteen heads' worth of information must be squeezed through a bottleneck. The model learns which information to preserve — the O projection weights are trained to keep what matters and discard what doesn't.

Note: V is deliberately not normalized (no "V Norm") because the model uses V's magnitude to control how much information flows through each token. Normalizing V would strip away that control.

---

8. Everything In One Place

Every operation in a transformer is one of the five we just learned:

Plus two nonlinear functions: softmax (converts scores to probabilities) and sigmoid (the gate in gated attention). That's the entire mathematical foundation of the attention mechanism.

---

9. Scaling Up: From 2D to 2048D

The operations we did with 2D and 3D vectors are identical to what happens inside Qwen3.6-35B-A3B — just with larger matrices:

to step through the real matrix shapes interactively.

---

Next: Ep12 — Attention Heads. We load Qwen3.6-35B-A3B, feed it a sentence, and pause inside the attention heads — watching the actual Q, K, V projections happen on real vectors.