Skip to content

ML Model Selection

Picking the right model is 10% algorithm and 90% honest measurement. This sheet is about the validation machinery: how to split data, estimate generalization, tune hyperparameters without fooling yourself, and decide between models on evidence.

Table of Contents

Installation / Setup

pip install scikit-learn    # or: uv add scikit-learn

All selection tooling lives in sklearn.model_selection.

Core Syntax

Data roles — three partitions, three jobs

Partition Used for Interaction allowed
train Fitting the model Everything
validation Choosing hyperparameters/models Tuning decisions only
test Final, one-time estimate Never — not even once

When you have little data, k-fold CV plays the role of the validation set; the test set stays untouched until the end.

Splits

from sklearn.model_selection import train_test_split

# plain split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# classification: keep class balance in both sides
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
                                                    random_state=42, stratify=y)
Argument Meaning
test_size Fraction or count of the holdout
stratify=y Preserve class proportions (classification)
random_state Reproducible shuffle
shuffle=True Default — disable only for ordered/time data

Cross-validation

from sklearn.model_selection import cross_val_score, KFold, StratifiedKFold

scores = cross_val_score(model, X_train, y_train, cv=5, scoring="accuracy")
print(scores, "mean:", scores.mean().round(3), "±", scores.std().round(3))
Splitter Use
KFold(n_splits=5, shuffle=True, random_state=0) Regression (shuffled)
StratifiedKFold(5, shuffle=True, random_state=0) Classification — preserves class ratios per fold
GroupKFold(n_splits=5) Samples group together (a user's rows must not leak across folds)
TimeSeriesSplit(n_splits=5) Ordered data — train only on the past
LeaveOneOut() Tiny datasets

Always shuffle KFold unless data is sequential.

Bias–variance: reading model complexity

Underfit (high bias)          Good fit              Overfit (high variance)
train ~ low, test ~ low       train high, test high   train very high, test drops
  • Rising train error → underfitting (model too simple).
  • Falling train but flat/falling test error → overfitting (model too complex, or not enough data).
  • Fix underfit: more features, more capacity, less regularization.
  • Fix overfit: more data, regularization, fewer features, simpler model, early stopping.

Common Operations

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

param_grid = {
    "n_estimators": [100, 300, 500],
    "max_depth": [None, 10, 20],
    "min_samples_leaf": [1, 4],
}                                        # 3×3×2 = 18 combos × 5 folds = 90 fits
search = GridSearchCV(RandomForestClassifier(random_state=0), param_grid,
                      cv=5, scoring="f1", n_jobs=-1, verbose=1)
search.fit(X_train, y_train)

Exhaustive over small grids; each combo is evaluated with CV, the best cv score wins, and best_estimator_ is refit on all of X_train.

from sklearn.model_selection import RandomizedSearchCV
import numpy as np

param_dist = {
    "n_estimators": [int(x) for x in np.linspace(50, 500, 10)],
    "max_depth": [None] + list(range(5, 30, 5)),
    "min_samples_leaf": [1, 2, 4],
    "max_features": ["sqrt", "log2"],
}
search = RandomizedSearchCV(RandomForestClassifier(random_state=0), param_dist,
                            n_iter=40, cv=5, scoring="f1", n_jobs=-1, random_state=0)
search.fit(X_train, y_train)

Random sampling from the distribution covers more ground per fit than grid — use it for larger spaces; give continuous params distributions (scipy.stats.uniform, randint) when available.

Nested CV — the fully honest number

from sklearn.model_selection import cross_val_score, KFold, GridSearchCV

outer = KFold(5, shuffle=True, random_state=0)
inner = KFold(3, shuffle=True, random_state=1)

scores = []
for tr_idx, va_idx in outer.split(X_train, y_train):
    search = GridSearchCV(model, param_grid, cv=inner, scoring="f1")
    search.fit(X_train.iloc[tr_idx], y_train.iloc[tr_idx])
    scores.append(search.score(X_train.iloc[va_idx], y_train.iloc[va_idx]))

print("nested CV F1:", np.mean(scores))     # unbiased estimate of the tuned pipeline

Nested CV reports how well the whole tuning procedure generalizes; the inner loop tunes, the outer loop measures.

Practical Examples

Example 1 — Honest workflow end-to-end

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import (train_test_split, cross_val_score,
                                     GridSearchCV)
from sklearn.metrics import classification_report

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
                                          random_state=42, stratify=y)

# 1. baseline
from sklearn.dummy import DummyClassifier
print("baseline:", cross_val_score(DummyClassifier(strategy="most_frequent"),
                                   X_tr, y_tr, cv=5).mean())

# 2. tune inside CV (test set still untouched)
search = GridSearchCV(RandomForestClassifier(random_state=0),
                      {"n_estimators": [100, 300], "max_depth": [None, 10]},
                      cv=5, scoring="roc_auc", n_jobs=-1)
search.fit(X_tr, y_tr)
print("best CV AUC:", search.best_score_, search.best_params_)

# 3. one final touch of the test set
print(classification_report(y_te, search.predict(X_te)))

Example 2 — Learning curves to diagnose

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve

train_sizes, train_s, test_s = learning_curve(
    search.best_estimator_, X_tr, y_tr, cv=5,
    train_sizes=np.linspace(0.2, 1.0, 5), scoring="roc_auc")

plt.plot(train_sizes, train_s.mean(axis=1), label="train")
plt.plot(train_sizes, test_s.mean(axis=1), label="CV")
plt.legend(); plt.xlabel("training examples"); plt.ylabel("AUC")

Wide train/CV gap at the right edge → add data or reduce complexity. Both curves flat and low → underfitting (model too weak for the problem).

Example 3 — Model comparison with repeated CV

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score, RepeatedStratifiedKFold

models = {
    "logistic": LogisticRegression(max_iter=2000),
    "rf": RandomForestClassifier(n_estimators=200, random_state=0),
    "svm": SVC(kernel="rbf"),
}
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
for name, model in models.items():
    scores = cross_val_score(model, X_tr, y_tr, cv=cv, scoring="roc_auc")
    print(f"{name:8s} AUC {scores.mean():.3f} ± {scores.std():.3f}")

Repeated CV shrinks the noise of a single 5-fold run; compare means ± std rather than one lucky number.

Common Patterns

Pattern: tuned pipeline inside a single CV object

param_grid = {
    "prep__num__imp__strategy": ["mean", "median"],
    "model__n_estimators": [200, 500],
    "model__max_depth": [None, 12],
}
search = GridSearchCV(pipe, param_grid, cv=5, scoring="roc_auc", n_jobs=-1)
search.fit(X_train, y_train)
best = search.best_estimator_     # preprocessing tuned together with the model

Pattern: when to trust which estimate

You want to know… Use
Is model A better than a baseline? CV score vs DummyClassifier/mean predictor
What hyperparameters? GridSearchCV/RandomizedSearchCV best_params_
How good is my procedure overall? Nested CV
Final reported number for stakeholders One evaluation on the untouched test set
Does model A beat B significantly? Repeated CV + paired test (e.g. Wilcoxon on fold scores)

Pattern: holdout as final referee

# after ALL tuning/selection decisions are locked:
final_score = search.score(X_test, y_test)

Tune against CV/validation as many times as you like; the test set is spent once.

Best Practices

  • The test set is sacred: split once, touch it once, at the very end.
  • Always stratify classification splits; GroupKFold when rows share a group; TimeSeriesSplit for time.
  • Compare against a baseline (majority class, mean, trivial model) before tuning anything.
  • Search over hyperparameters of the whole pipeline (incl. preprocessors), inside CV.
  • Prefer RandomizedSearchCV once the grid has more than ~dozens of combos.
  • Use n_jobs=-1 and prune grids early when scores plateau.
  • Report mean ± std of CV, not the best fold.
  • Scale features when comparing distance-based models; use the same preprocessing for every candidate.
  • Freeze random_state across candidates so differences are model signal, not seed noise.

Common Mistakes

Mistake Fix
Tuning on the test set repeatedly Test set is one-shot; use a validation split or CV for iteration
Searching hyperparameters, then reporting best_score_ as final best_score_ is optimistic — verify on held-out test data
Fitting the scaler on all data before CV Preprocessors inside the pipeline so each fold is clean
Stratified vs plain split confusion on imbalanced data Use stratify (or StratifiedKFold) for classification
Random shuffles across a user's repeated rows GroupKFold / GroupShuffleSplit
Shuffling time series before splitting TimeSeriesSplit — train on past only
Grid too dense → days of runtime Random search, coarser grid, more folds fewer combos
Comparing models on different splits Same CV object + same scoring for all candidates

Troubleshooting

Symptom Cause / Fix
ValueError: n_splits=... cannot be greater than the number of members Too few samples per class — reduce folds or use stratify carefully
Search runs forever Lower n_iter, add n_jobs=-1, simplify the grid
Best CV score much higher than test score Optimistic best_score_, leakage, or small data — check pipeline and use nested CV
All models score identically to baseline Check for a data bug or target leakage, then revisit features
Different runs give different CV means Missing random_state — fix seeds and shuffle
Huge RAM usage during search Fewer parallel jobs, smaller data subset, simpler model
Search picks the most complex option always Add regularization/complexity penalties into the grid; check data size

Official Documentation

Last Updated

2026-09-06 — Initial version.