Skip to content

ML Dimensionality Reduction

Fewer, denser features from many correlated ones. Dimensionality reduction fights the curse of dimensionality, denoises data, speeds training, and — via PCA — gives you the prettiest 2D plots in ML.

Table of Contents

Installation / Setup

pip install scikit-learn umap-learn    # or: uv add scikit-learn umap-learn
# t-SNE ships with scikit-learn; UMAP is a separate package
Tool Type Purpose
PCA Linear, deterministic Compression, denoising, features for models, plotting
TruncatedSVD Linear, sparse-friendly Same as PCA for sparse/text matrices
LDA (sklearn) Supervised linear Separability for labeled data
t-SNE Nonlinear, stochastic Visualization only — 2D/3D structure
UMAP Nonlinear Visualization + general embedding, faster/bigger data
NMF Nonneg. factorization Parts-based topics (text, spectra)
SparseRandomProjection Random linear Very wide data, speed over accuracy

Core Syntax

PCA — the workhorse

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)   # PCA is scale-sensitive — always scale
pca = PCA(n_components=2).fit(X_scaled)
Z = pca.transform(X_scaled)                     # n_samples × 2

print(pca.explained_variance_ratio_)            # variance per component
print(pca.explained_variance_ratio_.sum())      # total variance kept
Attribute Meaning
components_ Direction of each PC in original-feature space
explained_variance_ratio_ Share of total variance per PC (descending)
mean_ The mean vector subtracted during fitting
n_components_ How many were kept

Choosing how many components — explained variance

import numpy as np

pca = PCA().fit(X_scaled)                        # fit all
cum = np.cumsum(pca.explained_variance_ratio_)
k95 = int(np.searchsorted(cum, 0.95)) + 1        # components for 95% variance
print(f"{k95} components explain 95% of variance")

# elbow rule: keep components before the cumulative curve flattens
pca = PCA(n_components=k95).fit(X_scaled)

95% is a common default; for downstream models treat n_components as a hyperparameter and tune it with CV.

t-SNE / UMAP — visualization rules

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, random_state=0,
            learning_rate="auto", init="pca").fit_transform(X_scaled)

import umap
embed = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=0).fit_transform(X_scaled)

t-SNE/UMAP distances are not meaningful — only local neighborhoods and cluster groupings are. Both are stochastic: fix random_state and re-run to confirm structure is real. perplexity (t-SNE) ≈ 5–50, scale with data; n_neighbors (UMAP) ≈ 15–50.

LDA — supervised direction

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

lda = LinearDiscriminantAnalysis(n_components=2).fit(X_scaled, y)
Z = lda.transform(X_scaled)     # at most n_classes − 1 components

LDA maximizes class separation (uses labels) — for visualization of labeled data it often beats PCA; also usable as a supervised dimensionality pre-step.

Common Operations

Reduce-then-train

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

pipe = make_pipeline(StandardScaler(), PCA(n_components=20),
                     LogisticRegression(max_iter=2000))
print(cross_val_score(pipe, X, y, cv=5, scoring="roc_auc").mean())

Text matrix via TruncatedSVD

from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer

X_text = TfidfVectorizer(max_features=20000).fit_transform(corpus)
svd = TruncatedSVD(n_components=50, random_state=0).fit(X_text)
topics = svd.transform(X_text)     # dense 50-dim doc vectors (LSI)

Don't mean-center sparse matrices (destroys sparsity) — that's exactly why TruncatedSVD exists.

Interpret components

import pandas as pd

loadings = pd.DataFrame(
    pca.components_.T,
    columns=[f"PC{i+1}" for i in range(pca.n_components_)],
    index=feature_names,
)
print(loadings["PC1"].abs().sort_values(ascending=False).head(8))
# features with the largest |loading| drive that component

Practical Examples

Example 1 — Denoise images / embeddings before a model

from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# 784-dim MNIST digits -> 30 dims keeps ~95% variance
pipe = make_pipeline(StandardScaler(),
                     PCA(n_components=0.95),          # keep 95% automatically
                     RandomForestClassifier(random_state=0))
pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))

PCA(n_components=0.95) — pass a float in (0,1) to keep that fraction of variance; scikit-learn picks k for you.

Example 2 — Embedding visualization of clusters/labels

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE

Z = TSNE(n_components=2, random_state=0).fit_transform(X_scaled)
plt.scatter(Z[:, 0], Z[:, 1], c=y, cmap="tab10", s=8, alpha=0.7)
plt.title("t-SNE colored by class"); plt.show()

Uses: sanity-check that labeled classes separate, or eyeball unsupervised clusters before committing to them.

Example 3 — PCA as feature engineering for a linear model

from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV

pipe = make_pipeline(StandardScaler(), PCA(), Ridge())
grid = {"pca__n_components": [5, 10, 20, 50],
        "ridge__alpha": [0.1, 1.0, 10.0]}
search = GridSearchCV(pipe, grid, cv=5, scoring="neg_mean_squared_error")
search.fit(X_train, y_train)
print(search.best_params_)

Collinear predictors (e.g. correlated sensors, one-hot explosion) become orthogonal PCs — often stabilizing ridge/linear fits.

Common Patterns

Pattern: PCA in the pipeline only (no leakage)

# PCA must fit on training folds, like any other transformer.
# Putting it in the pipeline with cross_val_score / GridSearchCV does this.
pipe = make_pipeline(StandardScaler(), PCA(0.95), LogisticRegression(max_iter=2000))

Manual pca.fit(X) before the split is a leakage bug — the same sin as fitting a scaler on everything.

Pattern: decide by job

Goal Tool
Squeeze features for a model PCA / TruncatedSVD (n_components tunable)
Remove noise / collinearity PCA
Plot high-dim data UMAP (big), t-SNE (small), PCA (first look)
Maximize class separation (labeled) LDA
Topic/parts discovery NMF or TruncatedSVD on TF-IDF
Interactive exploration in the browser UMAP + plotly/altair

Pattern: compare original vs reduced via CV

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

results = {
    "raw": cross_val_score(make_pipeline(StandardScaler(), RandomForestClassifier(random_state=0)),
                           X, y, cv=5).mean(),
    "pca20": cross_val_score(make_pipeline(StandardScaler(), PCA(20),
                                           RandomForestClassifier(random_state=0)),
                             X, y, cv=5).mean(),
}
print(pd.Series(results))

Trees often do fine on raw features (they handle redundancy); PCA helps most when you move to linear/nearest-neighbor models.

Best Practices

  • Scale before PCA — unstandardized features let big-unit columns dominate components.
  • Treat n_components as a tunable hyperparameter (CV), with 95%-variance or an elbow as the starting guess.
  • Fit PCA inside the pipeline so validation folds never leak into components.
  • t-SNE/UMAP are for visualization: interpret groupings, not interpoint distances or axes.
  • random_state=0 on stochastic methods; rerun to confirm structure.
  • Keep perplexity (t-SNE) and n_neighbors (UMAP) in sensible ranges relative to n.
  • Use TruncatedSVD on sparse text matrices, never mean-centered PCA.
  • Inspect loadings when interpretability matters — PCA axes can be explained by a few features.
  • For labeled problems, try LDA: 2–3 components often separate classes better than PCA 2D.

Common Mistakes

Mistake Fix
PCA without scaling StandardScaler() first (unless all features share units)
Fitting PCA on full data pre-split Pipeline — fit per fold
Reading t-SNE distances as real Only local structure/grouping is meaningful
Using t-SNE to "reduce dims" for a model Its cost function preserves neighbors, not variance — use PCA/UMAP embeddings carefully, or keep t-SNE for plotting only
Forgetting random_state on t-SNE/UMAP Fix seeds; rerun to confirm
Mean-centering a sparse matrix Use TruncatedSVD
Judging k only by total variance Trade off variance vs downstream CV score
Treating PCA components as causal factors They're linear recombinations — check loadings before storytelling
Clustering on 2D t-SNE output Cluster in full space; t-SNE groups can be artifacts

Troubleshooting

Symptom Cause / Fix
explained_variance_ratio_ barely above 0 for later PCs Data nearly isotropic — PCA won't help; try feature selection
t-SNE gives a uniform blob perplexity too low/high, or genuinely no local structure
t-SNE runs forever Subset the data or switch to UMAP (n_neighbors)
PCA on sparse data errors / dense explosion Use TruncatedSVD or SparseRandomProjection
Components flip sign between runs Sign is arbitrary — fix by convention or svd_solver="full" + seed
Reduced features perform worse in CV PCA discarded signal — tune n_components down or skip reduction for that model
UMAP import fails pip install umap-learn (separate package)
Clusters in plot don't match classes Expected — embeddings preserve neighborhood, not class purity; check other metrics

Official Documentation

Last Updated

2026-09-06 — Initial version.