Skip to content

NLP Preprocessing

Raw text is messy; models are sensitive to what you feed them. This sheet defines the standard preprocessing pipeline — and when to stop. Modern transformers often prefer raw text; classical ML (TF-IDF, bag-of-words) needs every step. Knowing which is the difference between clean results and silent data loss.

Table of Contents

Installation / Setup

pip install nltk spacy            # or: uv add nltk spacy
python -c "import nltk; nltk.download('punkt_tab', quiet=True); nltk.download('stopwords', quiet=True); nltk.download('wordnet', quiet=True)"
python -m spacy download en_core_web_sm

Core Syntax

The preprocessing pipeline (step by step)

raw text
  → 1. lowercase
  → 2. remove HTML / URLs / emails
  → 3. remove punctuation & special chars (optional — depends on task)
  → 4. tokenize (split into words)
  → 5. remove stopwords (optional)
  → 6. stem or lemmatize (not both)
  → 7. rejoin tokens

Each step is a trade-off: you gain normality and reduce vocabulary size, but you lose information. The right level depends on the task.

The decision table: classical ML vs transformers

Step Classical ML (TF-IDF, BoW) Transformers (BERT, GPT)
Lowercase Almost always Only if model is uncased (bert-base-uncased)
Remove HTML/URLs Yes Yes
Remove punctuation Yes (or tokenize then filter) No — model handles it
Tokenize word_tokenize (NLTK) or str.split Use model's tokenizer (fast subword)
Stop words Yes — reduces dimensionality No — context words matter
Stem / Lemmatize Yes — reduces vocabulary No — subword tokenization handles this
Remove numbers Often No — numbers carry signal
Normalize unicode Yes Model tokenizer handles it

Rule of thumb: if your pipeline uses TfidfVectorizer or CountVectorizer, preprocess heavily. If it uses AutoModelForSequenceClassification, leave text as-is and let the tokenizer do the work.

Common Operations

Step 1: Lowercasing

text = text.lower()

Lowercasing is almost always safe for topic classification and search; it can destroy information for NER (proper nouns) and sentiment ("AMAZING" vs "amazing" has different emphasis).

Step 2: Remove noise

import re

def remove_noise(text):
    text = re.sub(r"<.*?>", " ", text)           # HTML tags
    text = re.sub(r"http\S+|www\.\S+", " ", text)  # URLs
    text = re.sub(r"\S+@\S+", " ", text)            # emails
    text = re.sub(r"\d+", " ", text)                  # numbers (optional)
    text = re.sub(r"[^\w\s]", " ", text)              # punctuation (optional)
    text = re.sub(r"\s+", " ", text).strip()          # collapse whitespace
    return text

Step 3: Tokenize

from nltk.tokenize import word_tokenize

tokens = word_tokenize(text.lower())
# or for speed: tokens = text.lower().split()

Step 4: Stop words

from nltk.corpus import stopwords

stop = set(stopwords.words("english"))
tokens = [t for t in tokens if t not in stop and t.isalpha()]

Custom stop words: always inspect the list first — "not", "no", "very" matter for sentiment analysis.

Step 5: Stem or lemmatize (pick one)

from nltk.stem import SnowballStemmer, WordNetLemmatizer

# stemming: fast, choppy
stemmer = SnowballStemmer("english")
stemmed = [stemmer.stem(t) for t in tokens]

# lemmatization: slower, readable (needs POS hints for best results)
lemmatizer = WordNetLemmatizer()
lemmatized = [lemmatizer.lemmatize(t) for t in tokens]
When to stem When to lemmatize
TF-IDF / bag-of-words — speed matters Text display or human readability
No POS tagger available POS tags are available or accuracy matters
Vocabulary reduction is the priority Stem is going to produce gibberish

Step 6: Rejoin

clean_text = " ".join(tokens)   # or lemmatized / stemmed

Practical Examples

Example 1 — Classical ML preprocessing pipeline

import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import SnowballStemmer

stemmer = SnowballStemmer("english")
stop = set(stopwords.words("english"))

def preprocess_classical(text):
    text = re.sub(r"<.*?>", " ", text)
    text = re.sub(r"http\S+", " ", text)
    text = re.sub(r"[^\w\s]", " ", text)
    tokens = word_tokenize(text.lower())
    tokens = [t for t in tokens if t.isalpha() and t not in stop]
    return " ".join(stemmer.stem(t) for t in tokens)

# use with TfidfVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(preprocessor=preprocess_classical, max_features=5000)
X = vectorizer.fit_transform(corpus)

Example 2 — Transformer preprocessing (minimal)

import re

def preprocess_transformer(text):
    text = re.sub(r"<.*?>", " ", text)     # HTML
    text = re.sub(r"http\S+", " ", text)   # URLs
    text = re.sub(r"\s+", " ", text).strip()
    return text

# then feed raw text to the model's tokenizer:
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
inputs = tokenizer(preprocess_transformer(text), truncation=True, padding=True)

Example 3 — Handle messy social-media text

import re

def clean_social(text):
    text = re.sub(r"@\w+", "[USER]", text)       # mentions → [USER]
    text = re.sub(r"#(\w+)", r"\1", text)          # #topic → topic
    text = re.sub(r"http\S+", "[URL]", text)       # URLs → [URL]
    text = re.sub(r"\bRT\b", "", text)              # remove "RT"
    text = re.sub(r"[\U0001F600-\U0001F64F]", " [EMOJI] ", text)  # emojis
    text = re.sub(r"(.)\1{2,}", r"\1\1", text)    # "loooove" → "loove"
    text = re.sub(r"\s+", " ", text).strip()
    return text

Example 4 — Sentiment preprocessing (preserve negation)

def preprocess_sentiment(text):
    text = re.sub(r"<.*?>", " ", text)
    text = re.sub(r"http\S+", " ", text)
    tokens = word_tokenize(text.lower())

    # keep negation words — they flip sentiment
    negation_keep = {"not", "no", "never", "neither", "nor", "hardly", "barely", "doesn't",
                     "don't", "didn't", "won't", "wouldn't", "couldn't", "shouldn't"}
    stop = set(stopwords.words("english")) - negation_keep

    tokens = [t for t in tokens if t.isalpha() and t not in stop]
    return " ".join(tokens)

Common Patterns

Pattern: a reusable preprocessing function with flags

def preprocess(text, lower=True, remove_numbers=False, remove_punct=True,
               stopwords=True, stem=True, min_len=2):
    if lower:
        text = text.lower()
    text = re.sub(r"<.*?>|http\S+", " ", text)
    tokens = word_tokenize(text)
    if remove_numbers:
        tokens = [t for t in tokens if t.isalpha()]
    if remove_punct:
        tokens = [t for t in tokens if t.isalnum()]
    if stopwords:
        stop = set(stopwords.words("english"))
        tokens = [t for t in tokens if t not in stop]
    if stem:
        tokens = [SnowballStemmer("english").stem(t) for t in tokens]
    tokens = [t for t in tokens if len(t) >= min_len]
    return " ".join(tokens)

Pattern: batch processing with progress

from tqdm import tqdm

cleaned = [preprocess(t) for t in tqdm(df["text"])]
df["clean_text"] = cleaned

Pattern: extract structured info before cleaning

def extract_features(text):
    return {
        "word_count": len(text.split()),
        "has_question": "?" in text,
        "has_url": "http" in text.lower(),
        "has_emoji": bool(re.search(r"[\U0001F600-\U0001F64F]", text)),
    }

Extract features from the raw text, then clean for the model — you want both.

Best Practices

  • Never preprocess before you understand the task. Some steps destroy signal (lowercasing proper nouns, removing "not" for sentiment).
  • Transformers: preprocess minimally (strip HTML/URLs); the model's tokenizer does the rest.
  • Classical ML (TF-IDF/BoW): preprocess aggressively — lowercase, tokenize, stopwords, stem/lemma.
  • Always test both stemming and lemmatization — sometimes stemming wins despite being "worse."
  • For sentiment: preserve negation ("not good" ≠ "good") and capitalization ("amazing" ≠ "AMAZING").
  • Extract features from raw text, clean for the model — both matter.
  • Use tqdm for large corpora — preprocessing is I/O and CPU-bound.
  • Keep your preprocessing function pure (no side effects) so it's testable and reusable.

Common Mistakes

Mistake Fix
Stemming before TF-IDF but the stems are meaningless Try lemmatization, or no stemming at all — compare CV scores
Removing "not" for sentiment analysis Use a custom stop list that keeps negation words
Lowercasing when NER is needed NER models expect original casing — don't lowercase
Applying stop-word removal to transformer input Stop words carry context for BERT/GPT — don't remove them
Removing numbers from a financial dataset Numbers carry signal — keep them, or bin them
Forgetting to handle HTML/URLs Always strip HTML tags and URLs first
Stemming/lemmatizing after tokenizing with a transformer Transformer tokenizers handle this internally — no stemming needed
Not using tqdm on 1M+ documents Progress bars are not optional — watch the ETA

Troubleshooting

Symptom Cause / Fix
Model accuracy drops after preprocessing You removed useful signal — try minimal preprocessing
Empty strings after cleaning Stop words + short-word filter removed everything — relax filters
Stemmer producing non-words Expected — stems are not meant to be readable
Slow preprocessing on large corpus Use spacy.pipe() or multiprocessing; avoid regex with backtracking
Unicode errors Normalize unicode first (unicodedata.normalize("NFKC", text))
Inconsistent token counts between train/test Your preprocessing is not deterministic — fix random state or use a Pipeline
URLs/emails not removed by regex The regex may not match all URL formats — use http\S+ or a URL parser

Official Documentation

Last Updated

2026-09-06 — Initial version.