Skip to content

Statistical Distributions

The shapes randomness takes. Recognize the distribution, and you know its mean/variance, how to sample it, and — crucially — where it appears in ML (losses, priors, generative models, noise assumptions).

Table of Contents

Installation / Setup

import numpy as np
from scipy import stats      # one class per distribution

Every distribution shares the same API: rvs, pdf/pmf, cdf, ppf, fit, plus .mean(), .var() on frozen objects.

Core Syntax

Discrete distributions

Distribution Parameters Mean / Var PMF intuition ML/data use
Bernoulli $p$ $p$ / $p(1-p)$ one coin flip binary labels; BCE = -log Bernoulli
Binomial $n$, $p$ $np$ / $np(1-p)$ count of $n$ flips # clicks, # successes
Poisson $\lambda$ $\lambda$ / $\lambda$ rare-event counts counts (GLM), arrivals, queueing
Geometric $p$ $1/p$ trials until first success retries, waiting times
Categorical $\pi_1..\pi_k$ — one draw among $k$ classification labels; CE loss

Continuous distributions

Distribution Parameters Mean / Var Shape Data science role
Uniform $a$, $b$ $\frac{a+b}{2}$ / $\frac{(b-a)^2}{12}$ flat priors (uninformative), init
Normal $\mu$, $\sigma$ $\mu$ / $\sigma^2$ bell noise, CLT, weight init, VAE latent
Log-normal $\mu$, $\sigma$ see note right-skewed prices, incomes, response times
Exponential $\lambda$ $1/\lambda$ / $1/\lambda^2$ decaying inter-arrival times; memoryless
Gamma $k$, $\theta$ $k\theta$ / $k\theta^2$ flexible waiting times, rainfall
Beta $\alpha$, $\beta$ $\frac{\alpha}{\alpha+\beta}$ [0,1] flexible probabilities, priors, Thompson sampling
Dirichlet $\alpha_1..\alpha_k$ — simplex topic models (LDA), categorical priors
Student's $t$ $\nu$ 0 (ν>1) heavier tails small samples, robust regression
Chi-squared $k$ $k$ / $2k$ skewed variance tests, GOF, sums of squares
F $d_1, d_2$ — right-skewed ANOVA, comparing variances
Laplace $\mu$, $b$ $\mu$ / $2b^2$ peaked tails robust regression = Laplace likelihood
Pareto $x_m$, $\alpha$ heavy tails power laws income, web traffic, "long tail"

Facts worth knowing: - Log-normal: $Y = e^X$ where $X \sim$ Normal. Mean $= e^{\mu + \sigma^2/2}$ (not the median!). Median $= e^\mu$. - Exponential is memoryless: $P(X > s+t \mid X > s) = P(X > t)$. - Beta(1,1) = Uniform(0,1); Beta is the conjugate prior for a Bernoulli/Binomial $p$. - Normal: 68–95–99.7 rule within $\mu \pm 1/2/3\sigma$. Standard normal: $\mu=0, \sigma=1$ (z-scores).

Key relationships

Relationship Meaning
Binomial$(1, p)$ = Bernoulli Bernoulli is the n=1 case
Binomial$(n,p) \to$ Poisson / Normal Poisson for large $n$ small $p$; Normal for large $n$
Sum of $k$ squared Normals Chi-squared with $k$ df
Ratio of two scaled Chi-squares F distribution
$t_\nu \to$ Normal as $\nu \to \infty$ t recovers normal with big samples
Exponential$(1/\theta)$ = Gamma$(1, \theta)$ special case
$\sum$ iid Normal still Normal (CLT exact here)

Common Operations

# Frozen object: name your distribution once
rv = stats.norm(loc=0, scale=1)
rv.rvs(100, random_state=0)      # sample
rv.pdf(0), rv.cdf(1.96), rv.ppf(0.975)   # density, probability, quantile

# Discrete
stats.binom(n=10, p=0.5).pmf(3)      # P(exactly 3 successes)
stats.poisson(mu=4).cdf(6)           # P(X <= 6) for count data
stats.beta(a=2, b=5).mean()          # 2/7

# Estimate parameters from data
loc, scale = stats.norm.fit(data)    # MLE-ish fit
a, b, loc2, scale2 = stats.beta.fit(data)   # 4-param fit for beta

Practical Examples

1. "Is my data normal?" — quick check

import numpy as np
from scipy import stats

x = rng.normal(0, 1, 1000)
stat, p = stats.normaltest(x)        # D'Agostino test
print("normal" if p > 0.05 else "not normal", p)

2. Thompson sampling for A/B (Beta posterior)

# after observing successes a_i, failures b_i on arm i:
a_arm, b_arm = 300, 700     # arm A
a_other, b_other = 330, 670  # arm B

rng = np.random.default_rng(0)
# sample each arm's true conversion rate from its Beta posterior
draw_a = rng.beta(a_arm, b_arm)
draw_b = rng.beta(a_other, b_other)
pick = "A" if draw_a > draw_b else "B"

3. Choose a distribution for count data (Poisson GLM intuition)

counts = rng.poisson(lam=5, size=200)
assert np.isclose(counts.mean(), counts.var(), rtol=0.2)   # Poisson: mean = var
# if var >> mean (overdispersion) -> Negative Binomial instead

4. Check model calibration against Normal residuals

resid = (y - model.predict(X))
print(stats.shapiro(resid))            # normality of residuals (see hypothesis-testing.md)

Common Patterns

# Sample from a mixture (e.g., two clusters)
k = rng.binomial(1, 0.5, 1000)
sample = np.where(k, rng.normal(0, 1, 1000), rng.normal(5, 1, 1000))

# QQ plot to eyeball a fit
import matplotlib.pyplot as plt
stats.probplot(x, dist="norm", plot=plt)
plt.show()

# Which distribution fits best? compare log-likelihoods
candidates = [stats.norm, stats.lognorm, stats.expon]
best = max(candidates, key=lambda d: d.fit(x).logpdf(x).sum() if hasattr(d.fit(x), 'logpdf') else -np.inf)
# (simpler: fit each and compare .nnlf(x) — negative log-likelihood, lower is better)

Best Practices

  • Match the data type to the distribution: counts → Poisson/NegBinomial; binary → Bernoulli/Binomial; continuous positive → Log-normal/Gamma; proportions → Beta.
  • Use log-likelihood to compare fits (negative log-likelihood .nnlf, lower = better), not eyeballing histograms.
  • Frozen objects (stats.norm(loc, scale)) avoid repeated-argument bugs.
  • Check mean vs variance as a cheap fingerprint (Poisson: equal; Normal: independent; NB: variance > mean).
  • Prefer robust options (Student-$t$ likelihood) when data has outliers — outliers are normal-distribution poison.

Common Mistakes

Mistake Consequence Fix
Using Normal for strictly positive/skewed data Negative predictions, bad fit Log-normal / Gamma / Poisson
PDF read as probability "Density > 1" confusion Integrate over an interval
Treating Beta params as counts naively Wrong posterior updates Beta posterior: α + successes, β + failures
Assuming mean = median for skewed families Biased summaries Median for skewed; mean only symmetric
df for chi2 vs t confused Wrong critical values chi2: sum of k squares; t: ν = n − 1
Fitting normal to heavy-tailed data Fragile estimates Student-t or Laplace; check QQ plot
Log-normal $\mu, \sigma$ treated as mean/std of $Y$ Wrong moments They parameterize $\ln Y$

Troubleshooting

Symptom Cause Fix
stats.xxx.fit returns 4 values for 2-param dist scipy's loc, scale shift Read dist.fit(data) docs per distribution
Sample moments mismatch formula Population vs sample Use ddof properly; sample size too small
Histogram looks log-normal Data positive & skewed Try lognorm.fit; compare log-likelihoods
Beta samples hit exactly 0/1 Boundary data Add smoothing; use mixture or inflated model
p-value always ~0 in normaltest Large n hypersensitive Combine with QQ plot; don't over-trust small deviations

Official Documentation

Last Updated

2026-09-06 — Initial version.