Skip to content

Probability for ML

The language of uncertainty: every ML model is a probability distribution, classification is conditional probability, and training is maximum likelihood in disguise.

Table of Contents

Installation / Setup

import numpy as np
from scipy import stats       # distributions, rvs, pdf, cdf

Core Syntax

Rules of probability

Rule Formula Meaning
Complement $P(A^c) = 1 - P(A)$ Not A
Union $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ A or B
Intersection (general) $P(A \cap B) = P(A \mid B) P(B)$ A and B
Independence $P(A \cap B) = P(A) P(B)$ Knowing A tells nothing about B
Conditional $P(A \mid B) = \frac{P(A \cap B)}{P(B)}$ A given B

Independence ⇒ uncorrelated, but not conversely. Conditional on a third variable, two variables can even become dependent (Simpson's-paradox territory).

Bayes' theorem — the central rule of ML

$$P(H \mid D) = \frac{P(D \mid H)\, P(H)}{P(D)} \qquad \text{posterior} \propto \text{likelihood} \times \text{prior}$$

Term Name ML meaning
$P(H \mid D)$ Posterior Belief after seeing data
$P(D \mid H)$ Likelihood How well hypothesis explains data
$P(H)$ Prior Belief before data (regularization!)
$P(D)$ Evidence Normalizer (often ignored)
  • MLE: pick $H$ maximizing $P(D \mid H)$ — what most training does.
  • MAP: pick $H$ maximizing $P(D \mid H)P(H)$ — MLE + prior (L2 penalty ≈ Gaussian prior on weights).

Law of total probability: for a partition $B_1, \dots, B_k$: $P(A) = \sum_i P(A \mid B_i) P(B_i)$.

Random variables

Type Probability object Conditions
Discrete $X$ PMF $p(x) = P(X = x)$ $\sum_x p(x) = 1$
Continuous $X$ PDF $f(x)$ $f(x) \ge 0$; $\int f = 1$; $P(X=x) = 0$!
Both CDF $F(x) = P(X \le x)$ Non-decreasing; $\to 1$ as $x \to \infty$

PDF values can exceed 1 — a density is not a probability; integrate it over a range to get one.

Expectation & variance

$$E[X] = \sum_x x\,p(x) \quad \text{or} \quad \int x f(x)\,dx \qquad \mathrm{Var}(X) = E[(X - E[X])^2] = E[X^2] - E[X]^2$$

Property Formula
Linearity (no independence needed!) $E[aX + bY] = aE[X] + bE[Y]$
Constant shift $\mathrm{Var}(aX + b) = a^2 \mathrm{Var}(X)$
Independent sum $\mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y)$
Std dev $\sigma_X = \sqrt{\mathrm{Var}(X)}$

Linearity of expectation is why linear regression needs no independence assumptions; variance is not linear.

Covariance & correlation

$$\mathrm{Cov}(X,Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y] \qquad \rho = \frac{\mathrm{Cov}(X,Y)}{\sigma_X \sigma_Y}$$

  • $\rho \in [-1, 1]$; $\pm 1$ = perfect linear relation; $0$ = no linear relation (curves can still hide there).
  • Covariance matrix $\Sigma_{ij} = \mathrm{Cov}(X_i, X_j)$: symmetric, positive semi-definite — feeds PCA, LDA, Gaussian models.

Joint, marginal, conditional distributions

$$p(x) = \sum_y p(x, y) \quad \text{(marginalize)} \qquad p(y \mid x) = \frac{p(x, y)}{p(x)}$$

  • Generative models learn $P(X, Y)$; discriminative models learn $P(Y \mid X)$ directly.
  • Naive Bayes: $P(Y \mid X) \propto P(Y)\prod_i P(X_i \mid Y)$ (assumes features independent given class).
  • Conditioning as modeling: $E[Y \mid X]$ is the regression function — the best predictor of $Y$ given $X$.

Laws of large numbers & CLT

Law Statement Use
LLN Sample mean $\to E[X]$ as $n \to \infty$ Why more data helps
CLT Mean of $n$ iid draws ≈ Normal(mean, $\sigma/\sqrt{n}$) Confidence intervals, hypothesis tests, noise in SGD

CLT says averages of almost anything look Gaussian for large $n$ — regardless of the underlying distribution (finite variance assumed).

Information theory (the loss functions)

Quantity Formula Meaning
Entropy $H(p) = -\sum_x p(x) \log p(x)$ Expected surprise; lower = more predictable
Cross-entropy $H(p, q) = -\sum_x p(x) \log q(x)$ Our classification loss!
KL divergence $D_{KL}(p | q) = \sum_x p(x) \log \frac{p(x)}{q(x)}$ Extra bits from using $q$ instead of $p$
  • $D_{KL} \ge 0$, equals 0 only when $p = q$; not symmetric, not a metric.
  • Training a classifier = minimizing cross-entropy = minimizing $D_{KL}$(true distribution, model).
  • Log-loss/BCE/categorical CE are all cross-entropy in disguise.

Common Operations

# Empirical mean/variance/covariance from samples
x = rng.normal(5, 2, size=10000)
x.mean(), x.var(), x.std()
np.cov(X.T, ddof=0)               # covariance matrix of columns
np.corrcoef(X.T)                  # correlation matrix

# Exact moments of a known distribution
from scipy import stats
rv = stats.norm(loc=5, scale=2)
rv.mean(), rv.var(), rv.std()
rv.cdf(6) - rv.cdf(4)             # P(4 < X <= 6)

# Simulate an event probability
p_est = (rng.normal(0, 1, 100000) > 1.96).mean()   # ~0.025

Practical Examples

1. Bayes for a spam-style classifier (binary)

# P(spam | "offer") = P("offer" | spam) P(spam) / P("offer")
p_spam = 0.3                                   # prior
p_word_given_spam = 0.6                        # likelihood
p_word = p_word_given_spam * p_spam + 0.1 * (1 - p_spam)   # total probability
p_spam_given_word = p_word_given_spam * p_spam / p_word
# 0.72

2. Demonstrate the CLT

means = [rng.exponential(scale=1.0, size=50).mean() for _ in range(5000)]
print(np.mean(means), 1 / np.sqrt(50))         # ~1.0 vs ~0.141 (SE = sigma/sqrt(n))
import matplotlib.pyplot as plt
plt.hist(means, bins=50, density=True)          # bell-shaped, centered at 1

3. Cross-entropy as a classification loss

q = np.array([0.7, 0.2, 0.1])                   # model's predicted probs
p = np.array([1.0, 0.0, 0.0])                   # one-hot truth
ce = -(p * np.log(q + 1e-12)).sum()             # 0.357
# Training minimizes exactly this (see Deep Learning loss functions)

Common Patterns

# Softmax turns logits into a probability vector (ML output layer)
def softmax(z):
    z = z - z.max()                             # numerical stability
    e = np.exp(z)
    return e / e.sum()

# Weighted average as expectation
mu = (values * weights).sum() / weights.sum()

# Simulate rather than derive (Monte Carlo)
mc_estimate = f(rng.normal(0, 1, size=100_000)).mean()

Best Practices

  • Always normalize: PMF/PDF must sum/integrate to 1; check softmax outputs.
  • Use log-space for products of probabilities (underflow): log-likelihoods, log-sum-exp.
  • Remember conditioning changes everything — $P(A \mid B)$ and $P(A)$ can differ wildly; Simpson's paradox is a cautionary tale.
  • Use the empirical rule of thumb: for most ML, "assume Gaussian for big averages, verify otherwise."
  • Prefer expected values over point samples when reasoning about algorithms (SGD = noisy estimate of the true gradient).
  • When in doubt about an event probability, simulate it (Monte Carlo) — numpy makes this trivial.

Common Mistakes

Mistake Consequence Fix
$P(A \mid B) = P(B \mid A)$ Wrong posterior (base-rate fallacy) Use Bayes with the prior
Adding probabilities of overlapping events Double counting $P(A \cup B)$ subtract intersection
Treating $E[X^2]$ as $E[X]^2$ Wrong variance $\mathrm{Var} = E[X^2] - E[X]^2$
Independence assumed for variance Wrong error bars Check dependence; use $\mathrm{Cov}$
Reading a PDF as a probability "Probability > 1" confusion Integrate over an interval
Optimizing raw likelihood instead of log Underflow Maximize $\log$ likelihood (monotonic)

Troubleshooting

Symptom Cause Fix
NaN in softmax/probability products Underflow/overflow Subtract max; log-space; logsumexp
Histogram doesn't look Gaussian Too few samples / heavy tails Increase $n$; check for infinite variance
Correlation near 0 but relation obvious Nonlinear relation Also plot; try rank correlation
Naive Bayes underperforms Correlated features Feature selection or a discriminative model

Official Documentation

Last Updated

2026-09-06 — Initial version.