Skip to content

ML Preprocessing

Models only see what you feed them. Preprocessing is where most real-world ML wins are won — and where careless code quietly leaks test information into training. Everything here fits inside a Pipeline so it stays leakage-free.

Table of Contents

Installation / Setup

pip install scikit-learn pandas    # or: uv add scikit-learn pandas

Preprocessing lives in sklearn.preprocessing (scalers/encoders), sklearn.impute (missing values), and sklearn.compose (ColumnTransformer).

Core Syntax

The leakage rule (read first)

Fit on training data only. A transformer must never be fit on the test set or on the full dataset before the split. When a transformer sits inside a Pipeline used with cross_val_score or GridSearchCV, scikit-learn fits it on each training fold automatically — that is why pipelines are not optional.

# WRONG — scaler sees test data through fit_transform on all of X
scaler = StandardScaler()
X_all = scaler.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_all, y, ...)

# RIGHT — scaler is fit on train folds only
from sklearn.pipeline import make_pipeline
pipe = make_pipeline(StandardScaler(), model)
pipe.fit(X_train, y_train)

Scaling — when and which

Distance-based models (SVM, kNN, logistic/ridge regression, PCA, KMeans) need comparable scales. Tree models do not.

Transformer Formula (per column) When to use
StandardScaler z = (x − μ) / σ Default; Gaussian-ish features, most models
MinMaxScaler (x − min) / (max − min) → [0,1] Known bounds, neural nets, image pixels
RobustScaler (x − median) / IQR Outlier-heavy features
MaxAbsScaler x / max|x| → [−1,1] Sparse data
QuantileTransformer rank → uniform/Gaussian Skewed features, no distribution assumption

Encoding — categoricals to numbers

Encoder Output When to use
OneHotEncoder One binary column per category Nominal categories, tree and linear models
OrdinalEncoder Integer per category Ordinal categories (low < med < high)
LabelEncoder Integer per class Target y only — never features
TargetEncoder Category → mean target High-cardinality nominal features (needs CV to avoid overfit)

OneHotEncoder details: handle_unknown="ignore" (survive unseen test categories), drop="first" or drop="if_binary" to avoid collinearity in linear models, sparse_output=False to get a dense array/DataFrame.

Imputation — missing values

Imputer Strategy When
SimpleImputer mean / median / most_frequent / constant Quick defaults; median robust to outliers
KNNImputer Nearest-neighbor average Strong correlations between features
IterativeImputer Multivariate regression chain MCAR-ish gaps, ample data (experimental)
pd.fillna (pre-model) Value, ffill/bfill Only for EDA/time-series prep — never in a fitted pipeline

Time series note: use ffill with group awareness for ordered data; don't use global mean on trends.

Common Operations

Scale + split correctly

from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, stratify=y)
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf", probability=True))
pipe.fit(X_train, y_train)
pipe.score(X_test, y_test)      # scaling fit on train only — done right

Inspect before you transform

import pandas as pd

df.info()                              # dtypes + non-null counts
print(df.isna().sum())                 # missing per column
print(df.select_dtypes("object").nunique())   # cardinality per categorical
print(df.describe())                   # ranges, means → outliers, scale issues

Date & text preprocessing

df["date"] = pd.to_datetime(df["date"])
df["year"] = df["date"].dt.year
df["month"] = df["date"].dt.month
df["days_since_last"] = (pd.Timestamp.now() - df["date"]).dt.days

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(max_features=5000, stop_words="english")   # text -> features

Practical Examples

Example 1 — Mixed numeric/categorical table → clean design matrix

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.read_csv("customers.csv")
X = df.drop(columns=["churned"]); y = df["churned"]

num_cols = ["tenure", "monthly_charges", "support_calls"]      # inspect dtypes first
cat_cols = ["contract", "payment_method", "region"]

num = Pipeline([("imp", SimpleImputer(strategy="median")),
                ("sc", StandardScaler())])
cat = Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                ("oh", OneHotEncoder(handle_unknown="ignore", drop="if_binary"))])

prep = ColumnTransformer([
    ("numeric", num, num_cols),
    ("categorical", cat, cat_cols),
], remainder="drop")            # or "passthrough" to keep other columns

X_clean = prep.fit_transform(X)          # only for exploration / one-shot jobs
X_train_clean = prep.fit_transform(X_train)   # always fit on the split

Example 2 — Skewed numeric feature → log transform inside pipeline

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import FunctionTransformer, StandardScaler
import numpy as np

log = FunctionTransformer(np.log1p, feature_names_out="one-to-one")
skewed = ["spend", "session_minutes"]

prep = ColumnTransformer([
    ("log_scale", make_pipeline(log, StandardScaler()), skewed),
    ("rest", "passthrough", ["tenure", "age"]),
])

np.log1p (log(x+1)) handles zeros; wrap it in a FunctionTransformer so it composes in pipelines.

Example 3 — Text + numeric in one pipeline

from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

prep = ColumnTransformer([
    ("text", TfidfVectorizer(max_features=3000), "review_body"),
    ("num", StandardScaler(), ["rating", "word_count"]),
])
pipe = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000))])
pipe.fit(X_train, y_train)

Common Patterns

Pattern: full preprocessing pipeline with a model

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder

pipe = Pipeline([
    ("prep", ColumnTransformer([
        ("num", SimpleImputer(strategy="median"), num_cols),
        ("cat", Pipeline([
            ("imp", SimpleImputer(strategy="most_frequent")),
            ("oh", OneHotEncoder(handle_unknown="ignore")),
        ]), cat_cols),
    ])),
    ("model", RandomForestClassifier(random_state=0)),
])
pipe.fit(X_train, y_train)        # CV-safe: folds refit 'prep' every time

Pattern: inspecting what the preprocessor produced

prep.fit(X_train)
names = prep.get_feature_names_out()          # e.g. ['numeric__tenure', 'categorical__contract_one_year']
Xt = prep.transform(X_train)
df_clean = pd.DataFrame(Xt.toarray() if hasattr(Xt, "toarray") else Xt, columns=names)

Pattern: preventing leakage in target encoding

from sklearn.model_selection import cross_val_score

# If you target-encode, do it inside CV: encode per fold using only the fold's train part.
# sklearn's TargetEncoder + a Pipeline + cross_val_score handles this correctly.
from sklearn.preprocessing import TargetEncoder
pipe = make_pipeline(TargetEncoder(), RandomForestClassifier(random_state=0))
print(cross_val_score(pipe, X, y, cv=5, scoring="roc_auc").mean())

Best Practices

  • Fit preprocessors only on training folds — pipeline or fit_transform(X_train).
  • Never fit_transform scalers/encoders on X before train_test_split (the #1 subtle bug).
  • Median imputation + RobustScaler for outlier-heavy data; standard otherwise.
  • handle_unknown="ignore" on OneHotEncoder — production data brings new categories.
  • Put drop="first"/"if_binary" for linear models; harmless for trees.
  • Keep raw columns aligned with ColumnTransformer — no fragile manual loops.
  • feature_names_out/get_feature_names_out() when you need the produced column names (feature importance, debugging).
  • Log/power-transform heavy-tailed numeric features before scaling.
  • When tuning SimpleImputer strategy, address it via prep__num__imp__strategy in GridSearchCV — tuning the transformer is legitimate.

Common Mistakes

Mistake Fix
StandardScaler().fit_transform(X) on the whole dataset Fit per fold: pipeline or after the split
Encoding with LabelEncoder on features (implies order) OneHotEncoder or OrdinalEncoder per type
Scaling sparse matrices with StandardScaler Use MaxAbsScaler, or densify deliberately
Imputing time series with the column mean ffill/interpolate along time, or group-wise
OneHot on a 10k-category column TargetEncoder, frequency encoding, or embeddings
Dropping rows with any NaN before modeling Impute — dropping shrinks data and hides patterns
Fitting TF-IDF on train+test text together In the pipeline; fit on train vocabulary only
pd.get_dummies then model — col mismatch at predict time OneHotEncoder with handle_unknown="ignore"

Troubleshooting

Symptom Cause / Fix
ValueError: Input contains NaN, infinity or a value too large Impute + handle infs (e.g. np.nan_to_num) before model
Name mismatch warning X has feature names ... mismatch fit/transform must see the same column set — use the pipeline end-to-end
OneHot gives ValueError: unknown category at predict handle_unknown="ignore"
CV scores differ wildly from a single split Fit/transform leakage outside the pipeline, or high variance — check both
Huge sparse output OneHot without drop, or sparse_output=True
TfidfVectorizer on a column of NaN Impute empty strings or drop those rows first
Scaler produced all zeros for a constant column Constant features carry no signal — drop or verify with nunique()==1

Official Documentation

Last Updated

2026-09-06 — Initial version.