Skip to content

ML Feature Engineering

The best model on weak features loses to a decent model on strong ones. This sheet catalogs the transformations that turn raw data into signal — and how to add them without leaking the target into training.

Table of Contents

Installation / Setup

pip install scikit-learn pandas    # or: uv add scikit-learn pandas

Feature engineering is pandas for exploration + sklearn.preprocessing/sklearn.feature_selection/sklearn.feature_extraction for repeatable pipeline steps.

Core Syntax

The two engineering worlds

World Tools Repeatable in pipeline?
Exploration (notebook) pandas df["new"] = ... ❌ — must be re-derived per fold
Production (pipeline) FunctionTransformer, ColumnTransformer, encoders ✅ — fits on train folds only

Rule: any feature that touches the target, or summarizes statistics, must live in a pipeline or be computed per-fold. Static transforms of a single row (x², log(x), age * tenure) are safe anywhere.

Common Operations

Derived & domain features

# ratios and products — the classics for tree and linear models alike
df["charges_per_gb"] = df["monthly_charges"] / df["data_gb"]
df["support_per_month"] = df["support_calls"] / df["tenure"].clip(lower=1)

# capture rate-like relationships
df["usage_ratio"] = df["minutes_used"] / df["minutes_allowed"]

# aggregations over repeated rows (per customer/entity)
group = events.groupby("customer_id")
df["n_events"] = group.size()
df["avg_session"] = group["session_min"].mean()
df["days_since_last"] = group["event_date"].max().rsub(pd.Timestamp("today")).dt.days

Good derived features encode domain knowledge: rates, shares, distances, gaps, and counts — things the raw columns hide.

Dates → signal

d = pd.to_datetime(df["date"])
df["year"] = d.dt.year
df["month"] = d.dt.month
df["dow"] = d.dt.dayofweek                 # 0 = Monday
df["is_weekend"] = d.dt.dayofweek >= 5
df["hour"] = d.dt.hour
df["days_since_ref"] = (d - pd.Timestamp("2020-01-01")).dt.days
df["season"] = ((d.dt.month % 12) // 3)    # quarter-ish bucket

Prefer cyclic-safe forms for linear models (sin/cos of month/hour) or let trees split the raw integer.

df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

Text → numbers

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(max_features=5000, ngram_range=(1, 2),
                        stop_words="english", min_df=2, sublinear_tf=True)
X_text = tfidf.fit_transform(corpus)        # in a pipeline it fits per fold

# cheap handcrafted text features (safe, row-local)
df["word_count"] = df["text"].str.split().str.len()
df["has_exclamation"] = df["text"].str.contains("!", regex=False)

Binning & rounding

df["age_bin"] = pd.cut(df["age"], bins=[0, 18, 35, 55, 120],
                       labels=["minor", "young", "mid", "senior"])

# for linear models, one-hot the bins; for trees, feed continuous values
from sklearn.preprocessing import KBinsDiscretizer
kbd = KBinsDiscretizer(n_bins=5, encode="onehot-dense", strategy="quantile")

Interaction features

df["tenure_x_charges"] = df["tenure"] * df["monthly_charges"]

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, interaction_only=True)   # x_i * x_j only

Trees build interactions implicitly by splitting; linear models need explicit ones — PolynomialFeatures (degree 2, interaction_only) is the systematic version.

Practical Examples

Example 1 — Engineer a churn dataset (pandas stage)

import pandas as pd, numpy as np

df = pd.read_csv("customers.csv")
df["signup_date"] = pd.to_datetime(df["signup_date"])

df["tenure_days"] = (pd.Timestamp("today") - df["signup_date"]).dt.days
df["charges_per_call"] = df["monthly_charges"] / df["support_calls"].clip(lower=1)
df["high_usage_low_pay"] = ((df["minutes_used"] > df["minutes_used"].median())
                            & (df["plan"] == "basic")).astype(int)

df["bill_year_1"] = df.groupby("customer_id")["bill"].transform(lambda s: s.iloc[:12].mean())
df["bill_trend"] = df.groupby("customer_id")["bill"].transform(
    lambda s: np.polyfit(np.arange(len(s)), s, 1)[0] if len(s) > 1 else 0)

Example 2 — Leakage-safe version (pipeline stage)

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier

def add_derived(X):
    X = X.copy()
    X["charges_per_gb"] = X["monthly_charges"] / X["data_gb"].clip(lower=0.1)
    X["support_per_month"] = X["support_calls"] / X["tenure"].clip(lower=1)
    return X

fe = FunctionTransformer(add_derived, feature_names_out="one-to-one")

prep = ColumnTransformer([
    ("fe", fe, ["monthly_charges", "data_gb", "support_calls", "tenure"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["contract"]),
])
pipe = make_pipeline(prep, RandomForestClassifier(random_state=0))
# cross_val_score(pipe, X, y, cv=5) — features recomputed per fold, no leakage

Example 3 — Target-aware encoding done safely

from sklearn.preprocessing import TargetEncoder
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import GradientBoostingClassifier

# mean-target encoding by category, fitted inside each CV fold
pipe = make_pipeline(TargetEncoder(), GradientBoostingClassifier(random_state=0))
print(cross_val_score(pipe, X, y, cv=5, scoring="roc_auc").mean())

Example 4 — Feature selection pipeline

from sklearn.feature_selection import SelectKBest, f_classif, RFE
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression

# univariate: keep 20 best by F-test
sel_uni = make_pipeline(prep, SelectKBest(f_classif, k=20), LogisticRegression(max_iter=2000))

# recursive: let a linear model rank features
sel_rfe = make_pipeline(prep, RFE(LogisticRegression(max_iter=2000), n_features_to_select=15),
                        LogisticRegression(max_iter=2000))
Selector Idea When
SelectKBest(f_classif/mutual_info) Univariate score per feature Quick filter, first pass
RFE Recursively drop weakest Linear/kernel models, interpretable count
SelectFromModel (RF/L1) Model-based importance threshold After fitting an importance model
VarianceThreshold Drop near-constant columns Cheap pre-filter, not target-aware
Permutation importance Drop-in measurement After training, to audit features

Common Patterns

Pattern: one table of raw data → ready matrix (repeatable)

raw columns
   │  FunctionTransformer (derived, row-local)
   │  ColumnTransformer
   │    ├─ numeric: impute → scale (optionally log first)
   │    ├─ categorical: impute → one-hot / target-encode
   │    ├─ text: TfidfVectorizer
   │    └─ datetime: derive → treat as numeric/categorical
   ▼
selectors (SelectKBest / RFE / SelectFromModel)
   ▼
model

Pattern: audit features after training

import pandas as pd
from sklearn.ensemble import RandomForestClassifier

model = pipe.fit(X_train, y_train)
importances = model.named_steps["model"].feature_importances_
names = model.named_steps["prep"].get_feature_names_out()
ranking = pd.Series(importances, index=names).sort_values(ascending=False)
print(ranking.head(15))

Zero-importance engineered features → drop them; a top feature that is a pure id or future leak → investigate.

Best Practices

  • Engineer from domain knowledge first (rates, gaps, aggregates), then let selection prune.
  • Keep derived features row-local in pipelines; entity-level aggregates belong to preprocessing jobs, not a model fitted on one split.
  • Never use target statistics (mean target by group, future values) without per-fold fitting.
  • Log/power-transform skewed numeric features before scaling for linear models.
  • Cap cardinality: one-hot ≤ ~50 categories; target-encode high-cardinality.
  • Add a VarianceThreshold/constant-column drop early; keep feature count sane vs sample size.
  • After feature selection, re-validate with CV — selection itself can overfit.
  • Store feature names (get_feature_names_out) so deployed models map input columns correctly.
  • If two features are perfectly correlated, drop one for linear models (collinearity), harmless for trees.

Common Mistakes

Mistake Fix
Computing mean/median aggregates on all rows, then splitting Aggregates leak future info — compute per fold or per time window
Target encoding by hand on the full dataset Pipeline + TargetEncoder (per-fold)
df.fillna(df["x"].mean()) on train+test combined Impute inside the pipeline
Feeding customer_id as a feature High-cardinality ids overfit — drop or aggregate
Using dates as raw integers Trees can split them, but derived recency/season features help
Creating 500 features for 200 rows Feature explosion — select aggressively or regularize
One-hot a high-cardinality categorical Target encode, or keep top-K + "other"
Selecting features on all data, then CV Selection must happen inside CV (nested)
Feature engineering only in the notebook, lost at deployment Reimplement as pipeline steps or shared functions

Troubleshooting

Symptom Cause / Fix
Pipeline errors at predict time: missing feature Train had a column predict lacks — consistent get_feature_names_out / same prep path
Engineered feature has zero importance It's redundant or noise — drop it
Huge memory after one-hot Sparse output, drop rare categories, or target-encode
Cross-fold scores worse after engineering Overfit features or leakage — audit with permutation importance
Timestamp features make CV leak Re-split by time or entity; features must not see the future
Model great on train, weak on test Engineering leaked the target or id-like features crept in
NaN appears after FunctionTransformer Division by zero — clip denominators, handle infs

Official Documentation

Last Updated

2026-09-06 — Initial version.