Skip to content

Hypothesis Testing

The structured way to ask "is this difference real or just noise?" — A/B tests, model comparisons, and feature-significance checks all reduce to hypothesis tests.

Table of Contents

Installation / Setup

import numpy as np
import pandas as pd
from scipy import stats

For regression-style tests (coefficient p-values, ANOVA tables) use statsmodels.

Core Syntax

The framework

Piece Meaning Example
Null $H_0$ "no effect" (assumed true) new design = old design
Alternative $H_1$ the effect you suspect new design ≠ old
Test statistic number computed from data $t$, $\chi^2$, $F$, $z$
p-value $P(\text{data this extreme} \mid H_0)$ small = evidence against $H_0$
Significance level $\alpha$ threshold for rejecting 0.05 (0.01 strict, 0.1 lenient)

Recipe: 1) state $H_0$/$H_1$, 2) pick $\alpha$, 3) compute statistic + p-value, 4) reject $H_0$ if $p < \alpha$, 5) interpret with effect size.

Errors & power

$H_0$ true $H_0$ false
Don't reject ✅ correct ❌ Type II ($\beta$)
Reject ❌ Type I ($\alpha$) ✅ power $= 1-\beta$
  • Type I = false positive (crying wolf). Type II = false negative (missing a real effect).
  • Increasing $n$ increases power; lowering $\alpha$ reduces Type I but raises Type II.

What a p-value is (and isn't)

  • ✅ p = probability, assuming $H_0$ is true, of seeing data as or more extreme than yours.
  • ❌ NOT the probability that $H_0$ is true.
  • ❌ NOT the probability the result happened by chance.
  • ❌ 1 − p is not "confidence the effect is real."
  • p < 0.05 ≠ important — with huge $n$, trivial effects get tiny p-values.

One- vs two-sided

  • Two-sided $H_1: \mu_A \ne \mu_B$ — default; use unless a direction is justified in advance.
  • One-sided $H_1: \mu_A > \mu_B$ — more power, but pre-register the direction (post-hoc one-sided tests are p-hacking).

Common Operations

Choosing the test

Question Test scipy Data
Mean of one group vs value One-sample $t$ stats.ttest_1samp(x, mu) normal-ish, $n$ ok
Two group means Two-sample $t$ stats.ttest_ind(a, b, equal_var=False) (Welch) independent, normal-ish
Same subjects, two times Paired $t$ stats.ttest_rel(before, after) dependent pairs
Two groups, non-normal Mann–Whitney U stats.mannwhitneyu(a, b) independent, any
Paired, non-normal Wilcoxon stats.wilcoxon(before, after) paired, any
3+ group means One-way ANOVA stats.f_oneway(a, b, c) normal-ish, equal-ish var
3+ groups, non-normal Kruskal–Wallis stats.kruskal(a, b, c) any
Two categorical vars Chi-square independence stats.chi2_contingency(table) counts table
Observed vs expected counts Chi-square GOF stats.chisquare(observed, expected) counts
Two proportions z-test for proportions statsmodels.stats.proportion.proportions_ztest counts
Correlation Pearson / Spearman stats.pearsonr(a, b) / stats.spearmanr(a, b) numeric
Normality (pre-check) Shapiro stats.shapiro(x) numeric
Equal variances (pre-check) Levene stats.levene(a, b, c) groups

Assumptions to check

  1. Independence of observations — design decision, not a test.
  2. Normality for $t$/ANOVA: Shapiro test + QQ plot; CLT rescues large $n$ ($\gtrsim 30$–50 per group).
  3. Equal variance for classic ANOVA/t: Levene test; use Welch variants otherwise (default in scipy for t).

Practical Examples

1. A/B test (two-sample t-test)

rng = np.random.default_rng(42)
control = rng.normal(50, 10, 500)
variant = rng.normal(52.5, 10, 500)

t_stat, p = stats.ttest_ind(control, variant, equal_var=False)
print(f"t={t_stat:.2f}, p={p:.4f}")          # p < 0.05 -> reject H0

# Always pair with effect size + CI
diff = variant.mean() - control.mean()
se = np.sqrt(control.var()/len(control) + variant.var()/len(variant))
ci = (diff - 1.96*se, diff + 1.96*se)
print("diff:", diff, "95% CI:", ci)

2. Paired before/after

before = rng.normal(100, 15, 60)
after = before + rng.normal(3, 6, 60)          # +3 improvement, noisy
stat, p = stats.ttest_rel(before, after)
print(p)                                        # paired: same units -> less noise

3. Chi-square on a contingency table

table = pd.crosstab(df["region"], df["churned"])     # counts
chi2, p, dof, expected = stats.chi2_contingency(table)
print(f"chi2={chi2:.2f}, p={p:.4f}")                 # p<0.05: region & churn related

4. ANOVA across 3 groups

groups = [df[df["plan"] == plan]["usage"].values for plan in ["free", "pro", "enterprise"]]
f_stat, p = stats.f_oneway(*groups)
print(f"F={f_stat:.2f}, p={p:.4f}")
# follow-up: which pairs differ? post-hoc Tukey (statsmodels.stats.multicomp.pairwise_tukeyhsd)

5. Proportion z-test

from statsmodels.stats.proportion import proportions_ztest
count = np.array([120, 140]); nobs = np.array([1000, 1000])
z, p = proportions_ztest(count, nobs)

Common Patterns

# Compact decision helper
def decide(p, alpha=0.05):
    return "reject H0" if p < alpha else "fail to reject H0"

# Multiple comparisons correction (FDR — standard for many tests)
from statsmodels.stats.multitest import multipletests
reject, p_adj, _, _ = multipletests(p_values, method="fdr_bh")

# Report format: statistic, df, p, effect size
# t(98) = 2.31, p = .023, d = 0.46

Best Practices

  • Pre-register $H_0$, $\alpha$, and sample size — decide the test before seeing the data.
  • Report effect sizes and CIs with every p-value; significance without magnitude is incomplete.
  • Use Welch's $t$-test (unequal variances) by default — robust and standard.
  • Correct for multiple comparisons whenever you run many tests (FDR/Bonferroni).
  • Check assumptions (normality, variance) before parametric tests; fall back to rank tests.
  • Power analysis before collecting data: tiny samples can't detect small effects.
  • For regression coefficients and ANOVA tables with proper SEs, prefer statsmodels.

Common Mistakes

Mistake Consequence Fix
"p = 0.03 means 97% chance $H_0$ false" Fundamental misreading p is under $H_0$ data probability
Running 100 tests at $\alpha=0.05$ ~5 false positives expected Correct (FDR) or pre-specify
Paired data tested as independent Wrong (usually inflated) p ttest_rel for pairs
One-sided test chosen after seeing data Inflated significance Two-sided, or pre-register
Normality violated, small $n$ Invalid p-values Mann–Whitney/Wilcoxon
Stopping when p < 0.05 ("optional stopping") Biased results Fix sample size in advance
Statistically significant ≠ practically Tiny effects declared wins Report effect size + business impact

Troubleshooting

Symptom Cause Fix
p exactly 0.0 (prints) Underflow with huge t Report p < 0.001; use log scale
chi2_contingency error on non-counts Proportions passed instead of counts Pass the count table (or multiply by n)
Conflicting normality verdicts Shapiro hypersensitive at large $n$ Trust QQ plot + CLT at scale
ANOVA significant but no pairwise test Only "some group differs" Post-hoc Tukey (pairwise_tukeyhsd)
p ~ 1 suspiciously Wrong direction or data duplicated Check signs; check test choice

Official Documentation

Last Updated

2026-09-06 — Initial version.