Skip to content

scikit-learn

The one toolkit for classical machine learning in Python. Nearly everything in scikit-learn is an estimator with the same fit/predict interface, so one mental model covers regression, classification, clustering, preprocessing, and selection.

Table of Contents

Installation / Setup

pip install scikit-learn        # or: uv add scikit-learn
python -c "import sklearn; print(sklearn.__version__)"

Built-in toy datasets for practice: load_iris, load_digits, load_wine, load_breast_cancer, fetch_california_housing (regression), make_classification / make_regression / make_blobs (synthetic).

Core Syntax

The estimator API — one pattern for everything

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(X_train, y_train)          # learn from data
y_pred = model.predict(X_test)       # predict labels
prob = model.predict_proba(X_test)   # class probabilities (classifiers)
score = model.score(X_test, y_test)  # default metric (accuracy / R²)
Estimator type Key methods Base class
Classifier fit, predict, predict_proba, score ClassifierMixin
Regressor fit, predict, score RegressorMixin
Transformer fit, transform, fit_transform TransformerMixin
Clusterer fit, fit_predict, labels_ ClusterMixin

Common constructor params: random_state (reproducibility), n_jobs (parallelism), verbose.

Key imports

# Datasets
from sklearn.datasets import load_iris, fetch_california_housing, make_classification

# Preprocessing
from sklearn.preprocessing import StandardScaler, MinMaxScaler, OneHotEncoder, LabelEncoder

# Split + validation
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV

# Models
from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge, Lasso
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor, GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA

# Pipelines + metrics
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.compose import ColumnTransformer
from sklearn.metrics import accuracy_score, precision_score, classification_report, confusion_matrix, mean_squared_error, r2_score
from sklearn.feature_selection import SelectKBest, f_classif
import joblib

Common Operations

train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)   # stratify for classification
  • stratify keeps class proportions — use for classification.
  • Always fix random_state so results are reproducible.

Mini-pattern: regression with pipeline + CV

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

pipe = make_pipeline(StandardScaler(), RandomForestRegressor(n_estimators=200, random_state=0))
scores = cross_val_score(pipe, X, y, cv=5, scoring="neg_mean_squared_error")
print("CV RMSE:", (-scores.mean()) ** 0.5)

Mini-pattern: classification with GridSearchCV

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

param_grid = {"n_estimators": [100, 300], "max_depth": [None, 10]}
search = GridSearchCV(RandomForestClassifier(random_state=0), param_grid,
                      cv=5, scoring="f1", n_jobs=-1)
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_)
test_acc = search.score(X_test, y_test)

Mini-pattern: clustering + PCA visualization

from sklearn.cluster import KMeans
from sklearn.decomposition import PCA

kmeans = KMeans(n_clusters=3, random_state=0, n_init=10).fit(X)
labels = kmeans.labels_
pca = PCA(n_components=2).fit_transform(X)

Mini-pattern: model persistence

import joblib

joblib.dump(model, "models/churn_rf.joblib")        # save (also the pipeline!)
loaded = joblib.load("models/churn_rf.joblib")      # load
loaded.predict(X_new)

Save the whole Pipeline, not a bare model — then predictions reuse the same preprocessing.

Practical Examples

Example 1 — End-to-end classification (numeric + categorical)

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
import joblib

df = pd.read_csv("customers.csv")
X = df.drop(columns=["churned"])
y = df["churned"]

num_cols = X.select_dtypes("number").columns.tolist()
cat_cols = X.select_dtypes("object").columns.tolist()

num_pipe = Pipeline([("impute", SimpleImputer(strategy="median")),
                     ("scale", StandardScaler())])
cat_pipe = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                     ("encode", OneHotEncoder(handle_unknown="ignore"))])

preprocessor = ColumnTransformer([
    ("num", num_pipe, num_cols),
    ("cat", cat_pipe, cat_cols),
])

pipe = Pipeline([("prep", preprocessor),
                 ("clf", RandomForestClassifier(random_state=0))])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
                                                    random_state=42, stratify=y)
pipe.fit(X_train, y_train)

print(classification_report(y_test, pipe.predict(X_test)))
joblib.dump(pipe, "churn_model.joblib")

Example 2 — Regression with preprocessing + tuning

from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import RandomizedSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error

X, y = fetch_california_housing(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

param_dist = {
    "gbr__learning_rate": [0.01, 0.05, 0.1],
    "gbr__max_depth": [3, 5],
    "gbr__subsample": [0.8, 1.0],
}
pipe = make_pipeline(StandardScaler(), GradientBoostingRegressor(random_state=0))
search = RandomizedSearchCV(pipe, param_dist, n_iter=10, cv=5, n_jobs=-1, random_state=0)
search.fit(X_train, y_train)
print("Best:", search.best_params_)
print("MAE:", mean_absolute_error(y_test, search.predict(X_test)))

Example 3 — Cluster customers

from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X[["tenure", "spend", "calls"]])
km = KMeans(n_clusters=4, random_state=0, n_init=10).fit(X_scaled)
df["segment"] = km.labels_            # add labels back to the original frame
print(df.groupby("segment")[["tenure", "spend"]].mean())   # interpret clusters

Common Patterns

The universal pipeline recipe

Pipeline(steps=[
    ("prep", preprocessor),     # ColumnTransformer: num + cat branches
    ("reduce", PCA(...)),       # optional: dimensionality reduction
    ("select", SelectKBest(...)),  # optional: feature selection
    ("model", RandomForestClassifier(...)),
])

Always: preprocessing → model in one pipeline so CV and test-time predictions never leak training statistics.

Complete pattern: CV-safe GridSearch over a pipeline

from sklearn.model_selection import GridSearchCV

param_grid = {
    "prep__num__impute__strategy": ["mean", "median"],
    "model__n_estimators": [100, 300],
    "model__max_depth": [None, 8],
}
search = GridSearchCV(pipe, param_grid, cv=5, scoring="roc_auc", n_jobs=-1)
search.fit(X_train, y_train)
best = search.best_estimator_        # refit on full X_train by default

<step>__<param> addresses params of named pipeline steps — the reason naming steps matters.

Pattern: learning curve / overfit check

from sklearn.model_selection import learning_curve
import numpy as np

sizes, train_scores, test_scores = learning_curve(
    pipe, X_train, y_train, cv=5, train_sizes=np.linspace(0.2, 1.0, 5))
print("train:", train_scores.mean(axis=1).round(2))
print("test :", test_scores.mean(axis=1).round(2))   # gap grows => overfitting

Best Practices

  • One Pipeline for prep + model; never fit preprocessors separately on full data (target leakage).
  • Use train_test_split(..., stratify=y) for classification, random_state everywhere.
  • Prefer cross_val_score over a single split for honest estimates.
  • Grid/random search inside CV (GridSearchCV) so tuning doesn't leak into the test set — leave X_test untouched until the end.
  • Scale before distance-based models (SVM, kNN, PCA, KMeans) — not needed for trees.
  • Pick metrics by the business problem (recall when false negatives are expensive; f1 for imbalance; MAE for interpretable error).
  • Save pipelines with joblib, not bare models.
  • Use n_jobs=-1 for tuning; cap tree depth to control memory and runtime.
  • Compare against a baseline (majority class / mean prediction) before celebrating.

Common Mistakes

Mistake Fix
Fitting scaler/encoder on the whole dataset before the split Fit only on training folds — inside a Pipeline
Searching hyperparameters on X_test (even once) Tune with nested or CV-based search; test once at the end
Not stratifying an imbalanced split stratify=y in train_test_split
Trees on unnormalized data fine, then scaling anyway (harmless but useless) Scaling only helps distance-based models
predict_proba on a regressor or SVC without probability=True Classifiers expose it; regressors predict values
Saving the model but re-fitting scaler at load time Save the whole pipeline
Ignoring class imbalance and reporting accuracy Report precision/recall/f1 or roc_auc
Different random_state each run → non-reproducible numbers Fix seeds everywhere

Troubleshooting

Symptom Cause / Fix
ValueError: could not convert string to float Categorical columns not encoded — use OneHotEncoder in the pipeline
ValueError: Input X contains NaN Impute with SimpleImputer before the model step
Warning: feature names mismatch Fitted on raw arrays vs DataFrame with names — fit and predict on the same format
GridSearchCV extremely slow Reduce grid, add n_jobs=-1, use RandomizedSearchCV with n_iter
ConvergenceWarning (SVC/logistic) Scale features or raise max_iter
Score on test is far below CV score Data leakage in preprocessing or train/test drift
n_init deprecated warning on KMeans Set n_init="auto" or an integer explicitly

Official Documentation

Last Updated

2026-09-06 — Initial version.