Skip to content

Linear Algebra for ML

The language of machine learning: every dataset is a matrix, every model output a vector, and PCA, least squares, embeddings, and transformers are all linear algebra in disguise.

Table of Contents

Installation / Setup

No math install needed — use NumPy:

import numpy as np

Core Syntax

Vectors

A vector $x \in \mathbb{R}^n$ is an ordered list of $n$ numbers — a single data point or a set of weights.

Concept Definition Notes
Addition / scaling $(x+y)_i = x_i + y_i$, $(cx)_i = c x_i$ Elementwise
Dot product $x \cdot y = \sum_i x_i y_i = x^\top y$ Measures alignment
Euclidean norm $|x|_2 = \sqrt{x^\top x}$ Length; default norm
L1 norm $|x|_1 = \sum_i \lvert x_i \rvert$ Sparsity (lasso)
Unit vector $x / |x|_2$ Direction only
Cosine similarity $\cos\theta = \frac{x \cdot y}{|x||y|}$ Embedding similarity
Orthogonal $x \cdot y = 0$ No shared direction 
np.dot(a, b)                # or a @ b
np.linalg.norm(a)           # L2
np.linalg.norm(a, ord=1)    # L1
a / np.linalg.norm(a)       # unit vector
cos = a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

Matrices

$A \in \mathbb{R}^{m \times n}$ maps $n$-dim input to $m$-dim output. A dataset with $N$ rows and $d$ features is $X \in \mathbb{R}^{N \times d}$.

Operation Definition Use
Matrix-vector product $(Ax)i = \sum_j Ax_j$ Linear model: $y = Xw$
Matrix product $(AB){ik} = \sum_j A$}B_{jk Composing transforms
Transpose $(A^\top){ij} = A$ Swap rows/cols
Identity $I$ $AI = A$
Trace $\mathrm{tr}(A) = \sum_i A_{ii}$ Sum of eigenvalues
Determinant $\det(A)$ Volume scale; zero ⇔ singular
Inverse $A^{-1}A = I$ "Divide" by a matrix
Rank # independent rows/cols True dimensionality

Key facts: $(AB)^\top = B^\top A^\top$; $A$ invertible ⇔ $\det(A) \neq 0$ ⇔ full rank ⇔ no zero eigenvalues.

Eigenvalues & eigenvectors

$$A v = \lambda v$$

For a square matrix $A$, an eigenvector $v$ only gets stretched (by $\lambda$), not rotated. $A^k v = \lambda^k v$ — powers live in the eigenvalues.

  • $\det(A) = \prod_i \lambda_i$, $\mathrm{tr}(A) = \sum_i \lambda_i$
  • Symmetric $A$ ($A = A^\top$): real eigenvalues, orthonormal eigenvectors — covariance matrices qualify.
  • Positive semi-definite ($x^\top A x \ge 0$ for all $x$): all $\lambda_i \ge 0$ — true for covariance and Gram matrices; guarantees convex quadratic forms.
  • Sparse/ill-conditioned? Watch the condition number $\kappa = \lambda_{\max}/\lambda_{\min}$ (large $\kappa$ = numerically unstable).
w, V = np.linalg.eig(A)     # eigenvalues w, eigenvectors as columns of V
# symmetric: np.linalg.eigh(A) is faster and numerically safer

Singular value decomposition (SVD)

$$A = U \Sigma V^\top$$

Every matrix (even non-square) factorizes into: $U$ (left singular vectors), $\Sigma$ (singular values, non-negative, sorted), $V^\top$ (right singular vectors).

  • $\mathrm{rank}(A)$ = number of nonzero singular values.
  • Truncated SVD keeps the top $k$ singular values — best rank-$k$ approximation of $A$ (Eckart–Young).
  • This is exactly what PCA / latent semantic analysis / word embeddings compute.
  • Pseudoinverse $A^+ = V \Sigma^{-1} U^\top$: the least-squares solution machine.
U, s, Vt = np.linalg.svd(A, full_matrices=False)
k = 2
A_k = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]     # best rank-2 approx

Other decompositions

Decomposition Form Use
LU $A = LU$ Solving linear systems
QR $A = QR$ Least squares, stability
Cholesky $A = LL^\top$ SPD matrices; sampling MVN; Kalman/GP
Eigendecomposition $A = V\Lambda V^{-1}$ Square, diagonalizable

Common Operations

X @ w                      # matrix-vector (linear model forward)
X.T @ X                    # Gram matrix (covariance unscaled)
np.linalg.solve(A, b)      # SOLVE, never A_inv @ b for this
np.linalg.inv(A)           # explicit inverse (rarely needed)
np.linalg.det(A), np.trace(A)
np.linalg.matrix_rank(X)
np.linalg.norm(A)          # Frobenius norm = sqrt(sum of sq)
np.linalg.lstsq(X, y, rcond=None)   # least-squares solution
np.linalg.pinv(X)          # pseudoinverse

Practical Examples

1. Cosine similarity between embeddings

def cos_sim(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

# RAG retrieval: score = cosine similarity of query vs document embeddings
scores = np.array([cos_sim(query_emb, doc_emb) for doc_emb in doc_embeddings])
top_k = np.argsort(scores)[-5:][::-1]

2. PCA in 4 lines (via SVD)

Xc = X - X.mean(axis=0)                 # center
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)
Z = Xc @ Vt.T                           # scores (projections)
W_pc = Vt[:2].T                         # top-2 principal directions
X_reduced = Xc @ W_pc                   # 2-D embedding for plotting
explained = (s**2) / (s**2).sum()       # variance explained per component

3. Least-squares regression in closed form

Xb = np.c_[np.ones(len(X)), X]              # design matrix with bias
w = np.linalg.pinv(Xb) @ y                  # = lstsq solution
# geometrically: y_hat = Xb w is the projection of y onto col(Xb)

Common Patterns

# Standardize -> covariance -> eigen -> PCA
cov = np.cov(Xc, rowvar=False)         # d x d covariance
w, V = np.linalg.eigh(cov)              # eigh for symmetric
proj = Xc @ V[:, ::-1][:, :2]           # keep top-2 by eigenvalue

# Linear layer as affine transform (DL)
z = X @ W.T + b

# Attention score = scaled dot product (transformers)
scores = Q @ K.T / np.sqrt(d_k)

Best Practices

  • Solve systems with np.linalg.solve/lstsq, never by computing $A^{-1}$ — faster and numerically stable.
  • Use eigh for symmetric matrices (covariance, Gram) — assumes and exploits symmetry.
  • Center data before PCA/SVD; scale features when units differ.
  • Think in ranks, not sizes — $\mathrm{rank}(X)$ tells you the true degrees of freedom and whether $X^\top X$ is invertible.
  • Check the condition number before inverting near-singular matrices; regularize (ridge adds $\lambda I$).
  • Dimension errors are the #1 bug: verify with X.shape and remember $(AB)^\top = B^\top A^\top$ when deriving gradients.

Common Mistakes

Mistake Consequence Fix
np.linalg.inv(X.T @ X) @ X.T @ y Slow + unstable for correlated features np.linalg.lstsq(X, y) or add ridge
Assuming $X^\top X$ invertible Failure with $N < d$ or collinear columns Regularize, reduce dims, use pinv
Forgetting to center in PCA First PC captures the mean Subtract column means first
eig on non-symmetric matrices in ML Complex eigenvalues appear Check symmetry → eigh
Mixing row vs column vectors Shape errors in @ Keep a convention; .reshape(-1, 1) for columns
Norm without axis on a matrix Whole-matrix norm np.linalg.norm(X, axis=1)

Troubleshooting

Symptom Cause Fix
LinAlgError: Singular matrix Zero determinant / collinear columns Regularize or drop redundant features
ValueError: matmul: shape mismatch Incompatible dims Confirm inner dims match; check transposes
Complex eigenvalues unexpectedly Asymmetric matrix or numerical noise Use eigh on symmetric; symmetrize $A + A^\top$
PCA components look wrong Uncentered/unscaled data X - mean; standardize features
Tiny/large norms in softmax/attention Poor scaling Divide by $\sqrt{d}$ (scaled dot-product)

Official Documentation

Last Updated

2026-09-06 — Initial version.