Skip to content

NLTK

The Natural Language Toolkit — the classical library for teaching and prototyping NLP in Python. NLTK gives you rule-based tokenizers, POS taggers, stemming, lemmatization, corpus readers, and VADER sentiment analysis. For production pipelines or large corpora, spaCy or Hugging Face are often faster; NLTK remains unmatched for educational breadth and quick experimentation.

Table of Contents

Installation / Setup

pip install nltk                  # or: uv add nltk
python -c "import nltk; nltk.download('punkt', quiet=True); nltk.download('stopwords', quiet=True); nltk.download('wordnet', quiet=True); nltk.download('averaged_perceptron_tagger', quiet=True); nltk.download('vader_lexicon', quiet=True); nltk.download('punkt_tab', quiet=True); nltk.download('averaged_perceptron_tagger_eng', quiet=True)"

Common downloads:

Download Purpose
punkt / punkt_tab Sentence + word tokenizers
stopwords Stop-word lists for 20+ languages
wordnet WordNet lemmatizer lexical database
averaged_perceptron_tagger / averaged_perceptron_tagger_eng English POS tagger
vader_lexicon Rule-based sentiment (VADER)
opinion_lexicon / movie_reviews / twitter_samples Sentiment / classification corpora

Core Syntax

Tokenization

from nltk.tokenize import sent_tokenize, word_tokenize

sentences = sent_tokenize("Hello world. How are you?")
# ["Hello world.", "How are you?"]

tokens = word_tokenize("The cat's in the hat!")
# ["The", "cat", "'s", "in", "the", "hat", "!"]
Tokenizer Type When to use
word_tokenize Punkt-based, English General-purpose word splitting
sent_tokenize Punkt-based, English Splitting paragraphs into sentences
regexp_tokenize Regex Custom tokenization rules (e.g. preserve hashtags)
TreebankWordTokenizer Penn Treebank conventions Matching the POS tagger's expectations

Stopwords

from nltk.corpus import stopwords

stop = set(stopwords.words("english"))    # ~179 words for English
tokens = [w.lower() for w in word_tokenize(text)]
filtered = [w for w in tokens if w.isalpha() and w not in stop]

Stop-word lists per language: stopwords.words("french"), "german", etc. Always check that the words removed are truly uninformative for your task (e.g. "not" is in NLTK's English stop list but is critical for sentiment).

Stemming

from nltk.stem import PorterStemmer, SnowballStemmer

porter = PorterStemmer()
snowball = SnowballStemmer("english")

print(porter.stem("running"))      # "run"
print(porter.stem("better"))       # "better"  (stems are often non-words!)
print(snowball.stem("running"))    # "run"
print(snowball.stem("studies"))    # "studi"  (Snowball = Porter v2)

Stemming chops suffixes — fast but imprecise; output may not be a real word.

Lemmatization

from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize("running", pos="v"))     # "run"
print(lemmatizer.lemmatize("better", pos="a"))       # "good"
print(lemmatizer.lemmatize("studies", pos="n"))      # "study"
POS tag Meaning Example
pos="n" Noun (default) studies → study
pos="v" Verb running → run
pos="a" Adjective better → good
pos="r" Adverb

Lemmatization needs POS hints to work well; stemming does not.

POS Tagging

from nltk import pos_tag
from nltk.tokenize import word_tokenize

tokens = word_tokenize("The quick brown fox jumps over the lazy dog")
tagged = pos_tag(tokens)
# [("The", "DT"), ("quick", "JJ"), ("brown", "NN"), ("fox", "NN"), ...]
Tag Meaning
NN / NNS / NNP Noun (singular / plural / proper)
VB / VBD / VBG / VBN Verb (base / past / gerund / past part.)
JJ / JJR / JJS Adjective (base / comparative / superlative)
RB / RBR / RBS Adverb
DT Determiner (the, a)
IN Preposition / subordinating conjunction
TO "to"
CC Coordinating conjunction (and, or)

Get the full tagset: nltk.help.upenn_tagset().

Common Operations

Sentiment Analysis (VADER)

from nltk.sentiment import SentimentIntensityAnalyzer

sia = SentimentIntensityAnalyzer()
scores = sia.polarity_scores("This movie was absolutely amazing!")
print(scores)
# {'neg': 0.0, 'neu': 0.46, 'pos': 0.54, 'compound': 0.6588}

# compound is the summary: > 0.05 positive, < -0.05 negative, else neutral

VADER handles social-media text well: emojis, emoticons, capitalization, punctuation emphasis ("AMAZING!!!"), and negation ("not bad" → positive).

Classical text classification with NaiveBayes

import nltk
from nltk.corpus import movie_reviews
from nltk.classify import NaiveBayesClassifier
from nltk.classify.util import accuracy

def extract_features(words):
    return {w: True for w in words}   # bag-of-words features

docs = [(extract_features(movie_reviews.words(fid)), cat)
        for cat in movie_reviews.categories()
        for fid in movie_reviews.fileids(cat)]

import random; random.seed(0); random.shuffle(docs)
cut = int(len(docs) * 0.8)
train, test = docs[:cut], docs[cut:]

clf = NaiveBayesClassifier.train(train)
print("accuracy:", accuracy(clf, test))
clf.show_most_informative_features(10)

Concordance and collocations

from nltk.text import Text

text = Text(word_tokenize(open("corpus.txt").read()))
text.concordance("model", width=80, lines=10)   # show context around a word
text.collocations(num=20)                         # frequent bigrams

# frequency distribution
from nltk import FreqDist
fd = FreqDist(w.lower() for w in word_tokenize(open("corpus.txt").read()))
fd.most_common(20)                                # top 20 words
fd.plot(30)                                       # matplotlib chart

Practical Examples

Example 1 — Clean text for a TF-IDF classifier

from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import SnowballStemmer

stemmer = SnowballStemmer("english")
stop = set(stopwords.words("english"))

def preprocess(text):
    tokens = word_tokenize(text.lower())
    tokens = [w for w in tokens if w.isalpha() and w not in stop]
    return [stemmer.stem(w) for w in tokens]

texts = ["The model ran beautifully on the test set",
         "Training was slow and loss never converged"]
print([preprocess(t) for t in texts])
# [['model', 'ran', 'beauti', 'test', 'set'], ['train', 'slow', 'loss', 'never', 'converg']]

Example 2 — POS-aware lemmatization

from nltk import pos_tag, word_tokenize
from nltk.stem import WordNetLemmatizer

tag_map = {"J": "a", "V": "v", "N": "n", "R": "r"}

lemmatizer = WordNetLemmatizer()
tokens = word_tokenize("The running dogs were chasing better things")
tagged = pos_tag(tokens)
result = [lemmatizer.lemmatize(w, tag_map.get(pos[0], "n")) for w, pos in tagged]
print(result)
# ['The', 'run', 'dog', 'be', 'chase', 'good', 'thing']

Example 3 — VADER sentiment on a review dataset

import pandas as pd
from nltk.sentiment import SentimentIntensityAnalyzer

sia = SentimentIntensityAnalyzer()

def sentiment_label(text):
    c = sia.polarity_scores(text)["compound"]
    if c > 0.05:  return "pos"
    if c < -0.05: return "neg"
    return "neutral"

df["sentiment"] = df["review_text"].apply(sentiment_label)
print(df["sentiment"].value_counts())

Common Patterns

Pattern: a reusable preprocessing pipeline

import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import SnowballStemmer

stemmer = SnowballStemmer("english")
stop = set(stopwords.words("english"))

def preprocess(text, stem=True, keep_stopwords=False):
    text = re.sub(r"<.*?>", " ", text)           # strip HTML
    text = re.sub(r"http\S+", " ", text)          # strip URLs
    tokens = word_tokenize(text.lower())
    tokens = [t for t in tokens if t.isalpha()]
    if not keep_stopwords:
        tokens = [t for t in tokens if t not in stop]
    if stem:
        tokens = [stemmer.stem(t) for t in tokens]
    return " ".join(tokens)

Pattern: use NLTK text objects for exploratory analysis

from nltk.text import Text
from nltk.tokenize import word_tokenize

words = word_tokenize(open("data.txt").read().lower())
text = Text(words)
text.common_contexts(["model", "data"], num=15)   # words sharing context
text.similar("fast")                                # substitution test
text.dispersion_plot(["model", "data", "train"])    # word location in corpus

Pattern: frequency-based feature selection

from nltk import FreqDist

all_words = [w.lower() for t in texts for w in word_tokenize(t) if w.isalpha()]
freq = FreqDist(all_words)
# keep the top 5000 words as vocabulary
vocab = [w for w, c in freq.most_common(5000)]
# then: features = {w: (w in tokens) for w in vocab}

Best Practices

  • Always download NLTK data explicitly at the start of your script (nltk.download); otherwise you get cryptic LookupErrors.
  • Use SnowballStemmer over PorterStemmer — it handles more edge cases and is faster.
  • Provide POS tags to WordNetLemmatizer when possible — the default (noun) produces poor results for verbs/adjectives.
  • VADER works well for social-media/review text; for formal text consider TextBlob or a transformer-based model.
  • For production pipelines, prefer spaCy (faster) or Hugging Face transformers (higher accuracy).
  • Keep the stop-word list small and task-specific: "not" and "no" matter for sentiment; "the" and "a" rarely do for topic classification.
  • word_tokenize is Punkt-trained on English; for other languages use the appropriate punkt download or a different tokenizer.

Common Mistakes

Mistake Fix
LookupError: Resource punkt_tab not found Run nltk.download('punkt_tab')
Stemming producing non-words ("beauti") Use lemmatization for readable output, or just accept stems for bag-of-words models
Lemmatizer returning the same word unchanged You likely forgot the POS tag argument — provide it
Stop-word removal stripping "not" and breaking sentiment Use a task-aware stop list
word_tokenize splitting contractions incorrectly It's the best general-purpose English tokenizer; accept the result or use regex tokenizer
Importing NLTK submodules before downloading data Download first, then import
Slow tokenization at scale NLTK is Python-only; switch to spaCy's C tokenizer for millions of documents

Troubleshooting

Symptom Cause / Fix
LookupError: Resource xyz not found nltk.download('xyz') first
word_tokenize errors on Unicode Ensure input is a string, not bytes
Stemmer producing empty strings Filter non-alpha tokens before stemming
POS tagger returns nonsensical tags Input is lowercased — POS taggers work better on original casing
SentimentIntensityAnalyzer gives all zeros Text is too short or all non-English
FreqDist.plot() fails in scripts Not in a notebook; call import matplotlib; matplotlib.use('Agg') before
Very slow on large corpora NLTK is not optimized for speed — switch to spaCy or use multiprocessing

Official Documentation

Last Updated

2026-09-06 — Initial version.