Skip to content

Calculus for ML

Derivatives are how models learn: gradient descent walks downhill, and backpropagation is the chain rule applied to millions of parameters at once.

Table of Contents

Installation / Setup

# Symbolic / automatic differentiation for checking math:
pip install sympy torch    # or: uv add sympy torch

NumPy gives numeric gradients by hand; PyTorch gives them for free via autograd.

Core Syntax

Derivative

The derivative $f'(x)$ is the rate of change — the slope of the tangent:

$$f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$$

Interpretation in ML: $\frac{\partial L}{\partial w}$ answers "if I nudge weight $w$ up a tiny bit, how much does the loss change, and in which direction?"

Differentiation rules

Rule Formula
Power $\frac{d}{dx} x^n = n x^{n-1}$
Constant multiple $\frac{d}{dx} c f = c f'$
Sum $(f + g)' = f' + g'$
Product $(fg)' = f'g + fg'$
Quotient $\left(\frac{f}{g}\right)' = \frac{f'g - fg'}{g^2}$
Chain rule $\frac{d}{dx} f(g(x)) = f'(g(x))\, g'(x)$

Common derivatives

Function Derivative Where you meet it
$x^n$ $n x^{n-1}$ polynomials
$e^x$ $e^x$ softmax, exponential
$\ln x$ $1/x$ log-likelihood, cross-entropy
$\sigma(x) = \frac{1}{1+e^{-x}}$ $\sigma(x)(1 - \sigma(x))$ logistic regression output
$\tanh(x)$ $1 - \tanh^2(x)$ RNN/LSTM gates, layer norms
$\log \sigma(x)$ (log-sigmoid) $1 - \sigma(x)$ BCE/log-loss derivative
Softmax $s_j$ $s_j(\delta_{jk} - s_k)$ classification head

Partial derivatives & gradients

For $f(x_1, \dots, x_n)$, the gradient stacks all partials:

$$\nabla f = \left(\tfrac{\partial f}{\partial x_1}, \dots, \tfrac{\partial f}{\partial x_n}\right)$$

  • The gradient points uphill (fastest increase); $-\nabla f$ points downhill — hence gradient descent.
  • Its magnitude tells sensitivity; zero gradient = stationary point (min / max / saddle).

Key vector results (worth memorizing):

$$\nabla_w (w^\top x) = x, \qquad \nabla_w \tfrac{1}{2}|Xw - y|^2 = X^\top(Xw - y), \qquad \nabla_W \tfrac{1}{2}|XW - Y|_F^2 = X^\top(XW - Y)$$

Jacobian & Hessian

Object Shape Meaning
Jacobian $J$ of $f: \mathbb{R}^n \to \mathbb{R}^m$ $m \times n$ All partials of a vector function
Hessian $H$ $n \times n$ Second derivatives; curvature
Gradient of scalar $f: \mathbb{R}^n \to \mathbb{R}$ $n$ Direction of steepest ascent
  • Hessian positive-definite at a stationary point ⇔ local minimum; negative-definite ⇔ local maximum; indefinite ⇔ saddle point.
  • Deep nets have many saddles — one reason optimization is hard beyond convexity.

Chain rule in backpropagation

For a computation graph $L = f(g(h(x)))$:

$$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial f}\cdot\frac{\partial f}{\partial g}\cdot\frac{\partial g}{\partial h}\cdot\frac{\partial h}{\partial x}$$

Backprop multiplies local derivatives backward through the graph (reverse-mode autodiff) — one forward pass + one backward pass gives gradients for every parameter.

import torch
x = torch.tensor([1.0, 2.0], requires_grad=True)
y = (x**2).sum()          # L = x1^2 + x2^2
y.backward()
x.grad                    # tensor([2., 4.]) = 2x  ✓

Taylor expansion

$$f(x) \approx f(a) + f'(a)(x-a) \quad \text{(linear)} \qquad f(x) \approx f(a) + f'(a)(x-a) + \tfrac{1}{2}f''(a)(x-a)^2 \quad \text{(quadratic)}$$

Used for: Newton's method, learning-rate analysis (too large a step overshoots the quadratic bowl), and local loss-surface intuition.

Convexity

$f$ is convex if for all $x, y$ and $\theta \in [0,1]$:

$$f(\theta x + (1-\theta)y) \le \theta f(x) + (1-\theta) f(y)$$

  • Convex ⇒ every local minimum is global. MSE and logistic loss are convex (in the weights).
  • Neural nets (ReLU etc.) are non-convex — we settle for good local minima.
  • Second-order test: convex ⇔ Hessian positive semi-definite everywhere.

Common Operations

import sympy as sp

x, w = sp.symbols("x w")
sp.diff(sp.log(1 + sp.exp(-x)), x)          # d/dx of -log(sigmoid(-x)) style
sp.diff((w * x - 1)**2, w)                  # 2*x*(w*x - 1) -> 2x(wx-1): matches 2*(Xw-y)*X scalar case

# Numeric gradient check of your analytic gradient
def f(w): return ((X @ w - y) ** 2).mean()
def grad_num(f, w, eps=1e-6):
    g = np.zeros_like(w)
    for i in range(len(w)):
        wp, wm = w.copy(), w.copy()
        wp[i] += eps; wm[i] -= eps
        g[i] = (f(wp) - f(wm)) / (2 * eps)
    return g

Practical Examples

1. One step of gradient descent by hand

X = np.array([[1.0], [2.0], [3.0]]); y = np.array([2.0, 4.0, 6.0])  # y = 2x
w = 0.0; lr = 0.1
for _ in range(3):
    grad = 2 * X.T @ (X @ w - y) / len(X)     # d/dw MSE (factor 2)
    w -= lr * grad
    print(w)          # 0.28 -> 0.4816 -> 0.6325... converging to 2 slowly

2. Chain rule through a logistic unit

z = X @ w                     # linear
p = 1 / (1 + np.exp(-z))      # sigmoid
# BCE loss gradient w.r.t. w:  dL/dw = X^T (p - y)   (chain rule collapses neatly!)
grad = X.T @ (p - y)

3. Autograd for a small net

import torch.nn as nn
model = nn.Linear(3, 1)
loss = nn.functional.mse_loss(model(X_t), y_t)
loss.backward()               # populates model.weight.grad
with torch.no_grad():
    model.weight -= 0.01 * model.weight.grad

Common Patterns

# Gradient descent loop skeleton (see optimization.md)
for epoch in range(epochs):
    pred = model(X)
    loss = ((pred - y) ** 2).mean()
    loss.backward()                    # chain rule, done for you
    opt.step(); opt.zero_grad()        # w <- w - lr * grad

# Closed-form sanity check for linear regression (zero-gradient solution)
w_star = np.linalg.lstsq(Xb, y, rcond=None)[0]

Best Practices

  • Keep the "nudge" interpretation — every gradient answers a concrete question about your loss.
  • Verify analytic gradients numerically (finite differences) before trusting a custom loss.
  • Let autodiff do the chain rule; your job is correct architecture and loss math, not hand-derived backprop.
  • Watch scale: sigmoid/tanh saturate → gradients vanish (use ReLU-family; see DL sheets); cross-entropy + softmax is numerically stable, never log-then-softmax separately.
  • Learning rate is a step, not a direction — the gradient supplies direction only.

Common Mistakes

Mistake Consequence Fix
Missing the chain rule on composed losses Wrong gradient sign/size Differentiate outer, then inner
Treating $\nabla f$ of a vector function as scalar Shape errors Track shapes: result matches parameter shape
Forgetting the factor 2 in MSE gradient Halved steps Derive from $\frac{1}{2}|e|^2$ form or test numerically
Sigmoid + raw MSE Vanishing gradients, slow training Use logits + cross-entropy
Assuming convexity for neural nets Expecting global optimum Accept good local minima; tune initialization
detach()-ing through a loss Zero gradient (silent!) Only detach targets/constants

Troubleshooting

Symptom Cause Fix
grad is None Tensor not requires_grad / detached Set requires_grad=True; don't detach the path
Loss frozen, gradient ~0 Saturation (sigmoid/tanh) or zero init ReLU init, better weight init, logits+CE
Gradient explodes Chain-rule product of big derivatives Gradient clipping, smaller LR, normalization
Gradient check fails Wrong analytic formula Check chain rule & factor; use central differences
NaN loss log(0) or division by 0 Add epsilon; use log_softmax/BCEWithLogits

Official Documentation

Last Updated

2026-09-06 — Initial version.