Skip to content

ML Evaluation

Accuracy is a terrible metric for most real problems. This sheet maps metrics to the decisions they inform — so you can answer "is this model good enough?" with evidence instead of vibes.

Table of Contents

Installation / Setup

pip install scikit-learn    # or: uv add scikit-learn

All metrics live in sklearn.metrics. In cross_val_score/GridSearchCV you pass a scoring= string; for details use sklearn.metrics directly.

Core Syntax

Confusion matrix — start here

              predicted
              neg    pos
actual neg  [ TN     FP ]
actual pos  [ FN     TP ]

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

cm = confusion_matrix(y_test, y_pred)     # rows = actual, cols = predicted
ConfusionMatrixDisplay(cm).plot()

Everything else in classification derives from the four cells:

Term Formula Meaning
Accuracy (TP+TN) / all Fraction correct — misleading when classes imbalanced
Precision TP / (TP+FP) Of those predicted positive, how many were right
Recall / Sensitivity TP / (TP+FN) Of actual positives, how many did we catch
Specificity TN / (TN+FP) Of actual negatives, how many did we correctly reject
F1 2·P·R / (P+R) Harmonic mean — balance when both matter
Fβ (1+β²)·P·R / (β²·P+R) Weighted: β>1 favors recall, β<1 favors precision

Which to watch — decision table

Problem Metric
Balanced classes, cheap errors accuracy
Rare disease/fraud: must catch positives recall (raise β in Fβ)
Spam filter: false positives annoy users precision
Both matter equally F1
Ranking / any threshold ROC-AUC or PR-AUC
Probability quality (not just ranking) log loss, Brier, calibration curve
Regression MAE / RMSE / R² / MAPE

Common Operations

Classification reports

from sklearn.metrics import (accuracy_score, precision_score, recall_score,
                             f1_score, classification_report)

print(classification_report(y_test, y_pred))   # per-class + macro/weighted
#              precision    recall  f1-score   support
#        0       0.91      0.93      0.92       430
#        1       0.78      0.74      0.76       120
#     accuracy                          0.89       550

Imbalanced data: watch the minority-class row and macro/weighted averages — overall accuracy hides the failure.

Probabilities, thresholds, curves

from sklearn.metrics import roc_auc_score, roc_curve, average_precision_score

probs = model.predict_proba(X_test)[:, 1]      # positive-class probability
print("ROC-AUC:", roc_auc_score(y_test, probs))          # ranking quality
print("PR-AUC :", average_precision_score(y_test, probs))  # better for imbalance

fpr, tpr, thresholds = roc_curve(y_test, probs)
Curve Axes Read it as
ROC TPR vs FPR How well positives separate from negatives at every threshold
PR Precision vs Recall Value of the model on the positive class (imbalance)
Calibration predicted prob vs actual freq Whether 0.7 means 70%

Thresholding — the button you actually press

# default threshold is 0.5, but the operating point is a business decision:
from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_test, probs)
t80 = thresholds[next(i for i, r in enumerate(recall) if r <= 0.8)]
y_t80 = (probs >= t80).astype(int)     # deploy this rule: catch 80% of positives

Always tune the threshold on a validation set, then confirm once on test.

Regression metrics

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

mae = mean_absolute_error(y_test, y_pred)              # same units, robust
rmse = mean_squared_error(y_test, y_pred) ** 0.5       # penalizes big misses
r2 = r2_score(y_test, y_pred)                          # % variance explained
mape = np.mean(np.abs((y_test - y_pred) / y_test))     # interpretable %, fails at y≈0
Metric Units Sensitive to Use when
MAE target units outliers equally You pay linearly per error
RMSE target units large errors (squared) Big misses hurt more
MAPE % near-zero targets Business speaks in % (watch zeros)
R² 0–1 (can go negative) fit relative to mean Communicating overall fit

Practical Examples

Example 1 — Churn model: choose the operating point

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import precision_recall_curve, classification_report
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
model = RandomForestClassifier(random_state=0).fit(X_train, y_train)
probs = model.predict_proba(X_test)[:, 1]

# retention budget: only contact people we are fairly sure will churn
precision, recall, thrs = precision_recall_curve(y_test, probs)
idx = next(i for i, p in enumerate(precision) if p <= 0.7)   # 70% precision point
rule = thrs[idx - 1]
print(f"threshold {rule:.2f} -> precision {precision[idx]:.2f}, recall {recall[idx]:.2f}")

flagged = probs >= rule
print(pd.Series(flagged).value_counts())
print(classification_report(y_test, flagged))

Example 2 — Medical-style recall target

from sklearn.metrics import recall_score

# must not miss positives; precision matters but recall is the contract
for t in [0.3, 0.4, 0.5, 0.6]:
    preds = (probs >= t).astype(int)
    print(f"t={t:.1f}  recall={recall_score(y_test, preds):.3f}  "
          f"precision={precision_score(y_test, preds):.3f}")
# pick smallest t meeting recall ≥ 0.95, then confirm on the test set

Example 3 — Calibration check (are probabilities real?)

from sklearn.calibration import calibration_curve

fraction_pos, mean_pred = calibration_curve(y_test, probs, n_bins=10)
for fp, mp in zip(fraction_pos, mean_pred):
    print(f"predicted ~{mp:.2f}  ->  actually {fp:.2f}")

# diagonal line = perfectly calibrated; systematic bow → CalibratedClassifierCV
from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, cv=5).fit(X_train, y_train)

Common Patterns

Pattern: pick the metric from the cost asymmetry

Cost matrix:  catching a churner saves $X; a wasted call costs $Y.
Precision/recall aren't aesthetic choices — they encode those costs.
False-negative-heavy cost  → maximize recall at a precision floor.
False-positive-heavy cost  → maximize precision at a recall floor.

Pattern: compare candidate models fairly

from sklearn.model_selection import cross_val_score

results = {}
for name, model in candidates.items():
    results[name] = cross_val_score(model, X, y, cv=5,
                                    scoring="roc_auc").mean()
print(pd.Series(results).sort_values(ascending=False))

Pattern: always pair the headline metric with context

summary = {
    "model": "rf_v3", "auc": 0.92, "threshold": 0.45,
    "precision@thr": 0.71, "recall@thr": 0.80,
    "flagged_pct": 0.18, "validation": "CV 5-fold, n=5_000",
}

A metric without the threshold, dataset size, and split description is unverifiable.

Best Practices

  • Report per-class numbers when classes are imbalanced — never accuracy alone.
  • Pick the metric from the business cost, not from what's easy to print.
  • Use predict_proba + threshold decisions rather than bare predict().
  • If you threshold-tune, do it on validation; report the final threshold's metrics on test.
  • For ranking (search, scoring leads), use ROC-AUC/PR-AUC; for decisions, use precision/recall at the operating point.
  • Check calibration when probabilities feed downstream decisions or are shown to users.
  • Regression: report MAE and RMSE (robustness + big-error view); include units.
  • Compare against a baseline metric (majority class accuracy, mean-MAE) every time.
  • Store the exact evaluation config (split seed, scoring, threshold) with results.

Common Mistakes

Mistake Fix
Accuracy on 95/5 data (95% "great") Report precision/recall/F1 or AUC on the minority class
Threshold tuned on the test set Thresholds belong on validation; test confirms once
Choosing the threshold that maximizes accuracy Choose by cost: precision vs recall trade-off
Treating AUC 0.9 as "the model is great" for decisions AUC ranks; check precision@the threshold you'll deploy
Averaging raw F1 per class without weights Use average="macro"/"weighted" deliberately
MAPE on data with zeros Use MAE/RMSE or filter near-zero targets
Confusing precision & recall in reports TP/(TP+FP) vs TP/(TP+FN) — read the confusion matrix
Reporting CV best_score_ as production expectation Report on held-out test data
Comparing models trained on different feature sets Same pipeline, same CV, same scoring for all

Troubleshooting

Symptom Cause / Fix
F1 lower than you expect despite high accuracy Imbalance — look at minority-class F1 and PR-AUC
Precision and recall both drop when threshold rises Contradiction — check you're comparing the right direction / same fold
AUC ≈ 0.5 despite "good" accuracy Probably imbalance + class prior; use PR-AUC / log loss too
Log loss explodes Model very confident and wrong — check for label leaks or calibration
RMSE ≫ MAE A few huge errors dominate — investigate outliers
R² negative Model worse than predicting the mean — data/target bug or wrong model class
Test metric far below CV Leakage or optimistic tuning — revisit the pipeline, consider nested CV

Official Documentation

Last Updated

2026-09-06 — Initial version.