Skip to content

Tokenizers

The engine behind every Hugging Face model. Tokenizers (the Rust-backed library) gives you fast subword tokenization: BPE, WordPiece, Unigram — each a way to split rare words into known pieces. Understanding this layer explains why bert tokenizes differently from gpt2, and how to train a tokenizer for your own vocabulary.

Table of Contents

Installation / Setup

pip install tokenizers                    # or: uv add tokenizers

The library is Rust-backed — very fast. The transformers library wraps it; tokenizers is for standalone use or training custom tokenizers.

Core Syntax

Three subword algorithms

Algorithm How it works Used by
BPE (Byte-Pair Encoding) Merge most frequent character pairs iteratively GPT-2, RoBERTa, LLaMA
WordPiece Like BPE but merges maximize language model likelihood BERT, DistilBERT, Electra
Unigram Start large, prune least useful subwords (opposite of BPE) T5, XLNet, ALBERT

All three: frequent words stay whole, rare words get split into known subwords.

From transformers tokenizer to raw tokenizers

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
print(type(tok))   # PreTrainedTokenizerFast (wraps tokenizers library)

# access the underlying fast tokenizer
backend = tok.backend_tokenizer
print(type(backend))  # <tokenizers.Tokenizer>

Encode and decode

from tokenizers import Tokenizer
from tokenizers.models import WordPiece

tokenizer = Tokenizer(WordPiece(vocab_file="vocab.txt", unk_token="[UNK]"))
output = tokenizer.encode("Hello, this is a test!")
print(output.ids)         # [101, 7592, 1010, 2023, 2003, 1037, 3231, 999, 102]
print(output.tokens)      # ['[CLS]', 'hello', ',', 'this', 'is', 'a', 'test', '!', '[SEP]']
print(output.attention_mask)  # [1, 1, 1, 1, 1, 1, 1, 1, 1]

Decode

tokenizer.decode([7592, 1010, 2023])    # "hello, this"
tokenizer.decode([101, 7592, 102], skip_special_tokens=True)  # "hello"

Common Operations

Encoding details

encoding = tokenizer.encode("Hello world!", add_special_tokens=True)
print(encoding)
# Encoding:
#   ids: [101, 7592, 2088, 999, 102]
#   tokens: ['[CLS]', 'hello', 'world', '!', '[SEP]']
#   attention_mask: [1, 1, 1, 1, 1]
#   type_ids: [0, 0, 0, 0, 0]
#   offsets: [(0,0), (0,5), (6,11), (11,12), (0,0)]
Field Meaning
ids Token IDs fed to the model
tokens Human-readable subword tokens
attention_mask 1 for real tokens, 0 for padding
type_ids Segment IDs (for sentence-pair tasks like BERT)
offsets (start, end) character positions for alignment
word_ids Which original word each token came from

Truncation and padding

from tokenizers import Encoding

# truncation
tokenizer.enable_truncation(max_length=128, strategy="longest_first")

# padding
tokenizer.enable_padding(length=128, pad_id=0, pad_token="[PAD]")

# encode batch
batch = tokenizer.encode_batch(["Hello world", "A longer sentence for testing"])
print(batch[0].attention_mask)   # [1, 1, 1, 1, 1, 0, 0, ...]  (padded)
Truncation strategy Behavior
"longest_first" Remove tokens from the longest sequence first
"only_first" Truncate only the first sequence
"only_second" Truncate only the second sequence

Pre-tokenization (before subword split)

from tokenizers.pre_tokenizers import Whitespace, ByteLevel, Digits

# whitespace split (BERT-style)
tokenizer.pre_tokenizer = Whitespace()

# GPT-2 style: splits on whitespace + byte-level
tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=False)

# split digits: "price123" -> ["price", "1", "2", "3"]
tokenizer.pre_tokenizer = Digits(individual_digits=True)

Normalization (before pre-tokenization)

from tokenizers.normalizers import NFD, Lowercase, Strip, Sequence, BertNormalizer

# BERT-style: lowercasing + accent stripping + unicode normalization
tokenizer.normalizer = Sequence([NFD(), Lowercase(), Strip()])

# for multilingual: NFD strips accents, useful before lowercasing
tokenizer.normalizer = Sequence([NFD(), StripAccents(), Lowercase()])

Post-processing (after tokenization)

from tokenizers.processors import TemplateProcessing

# BERT: add [CLS] at start and [SEP] at end
tokenizer.post_processor = TemplateProcessing(
    single="[CLS] $A [SEP]",
    pair="[CLS] $A [SEP] $B:1 [SEP]:1",
    special_tokens=[("[CLS]", 1), ("[SEP]", 2)],
)

# GPT-2: just an EOS token at end
tokenizer.post_processor = TemplateProcessing(
    single="$A [EOS]",
    special_tokens=[("[EOS]", tokenizer.token_to_id("[EOS]"))],
)

Practical Examples

Example 1 — Train a BPE tokenizer from scratch

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

# 1) create tokenizer with no vocab yet
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

# 2) train
trainer = BpeTrainer(
    vocab_size=30000,
    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
    min_frequency=2,
)
files = ["corpus_part1.txt", "corpus_part2.txt"]   # one file path per corpus
tokenizer.train(files, trainer)

# 3) save and use
tokenizer.save("bpe_tokenizer.json")

loaded = Tokenizer.from_file("bpe_tokenizer.json")
print(loaded.encode("uncommon subword tokenization").tokens)

Example 2 — Train a WordPiece for a domain-specific corpus

from tokenizers import Tokenizer
from tokenizers.models import WordPiece
from tokenizers.trainers import WordPieceTrainer
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

trainer = WordPieceTrainer(
    vocab_size=20000,
    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]"],
    min_frequency=2,
)
tokenizer.train(["medical_notes.txt"], trainer)

Example 3 — Align tokenized output to original text

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
encoding = tokenizer("New York City is big", return_offsets_mapping=True, return_tensors="pt")
offsets = encoding["offset_mapping"][0]

text = "New York City is big"
for i, token_id in enumerate(encoding["input_ids"][0]):
    start, end = offsets[i].tolist()
    print(f"{tokenizer.decode([token_id]):12s} -> '{text[start:end]}'")
# [CLS]        -> ''
# new          -> 'New'
# york         -> 'York'
# city         -> 'City'
# is           -> 'is'
# big          -> 'big'
# [SEP]        -> ''

Common Patterns

Pattern: vocabulary inspection

vocab = tokenizer.get_vocab()
print(f"vocab size: {len(vocab)}")
print(list(vocab.items())[:10])    # first 10 token->id pairs

# find what tokens encode to
encoding = tokenizer.encode("hello world")
for token, id in zip(encoding.tokens, encoding.ids):
    print(f"{token:15s} -> {id}")

Pattern: tokenize pairs (sentence similarity)

from tokenizers import Tokenizer
from tokenizers.models import WordPiece
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.processors import TemplateProcessing

tokenizer = Tokenizer(WordPiece(vocab_file="vocab.txt"))
tokenizer.pre_tokenizer = Whitespace()
tokenizer.post_processor = TemplateProcessing(
    single="$A [SEP] $B",
    pair="$A [SEP] $B:1 [SEP]:1",
    special_tokens=[("[SEP]", tokenizer.token_to_id("[SEP]"))],
)

encoding = tokenizer.encode("Hello", "World")
# encoding.ids includes [CLS] ... [SEP] ... [SEP]

Pattern: fast vs slow tokenizers

from transformers import AutoTokenizer

slow = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=False)
fast = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)

# both produce the same IDs; fast is 5-20x faster

Always use the fast tokenizer (the default in transformers) unless you have a specific reason not to.

Best Practices

  • Always use the fast tokenizer (PreTrainedTokenizerFast) — it's the default; slow is only for compatibility.
  • When training a new tokenizer, start with a representative corpus — the vocabulary reflects your data.
  • Use unk_token="[UNK]" for WordPiece; BPE and Unigram can produce byte-level fallback for unknowns.
  • BPE (GPT-2/LLaMA) = no word-boundary assumption; WordPiece (BERT) = word boundaries matter; Unigram (T5) = probabilistic pruning.
  • offsets are critical for token classification (NER, POS) — align tokens back to character positions.
  • When training a tokenizer for a transformer fine-tune, use the model's tokenizer_class and save_pretrained to save alongside the model.
  • For multilingual text: prefer NFD normalization + StripAccents in the normalizer.

Common Mistakes

Mistake Fix
Confusing tokenizer.tokenize() (strings) with tokenizer.encode() (IDs) tokenize() returns strings; encode() returns Encoding with ids + tokens
Training a tokenizer with too-small a corpus Minimum ~1M tokens for stable subword vocab; smaller = bad coverage
Forgetting special tokens in the vocab Include [UNK], [CLS], [SEP], [PAD], [MASK] for BERT-style models
Using use_fast=False without a reason Always use the fast tokenizer — it's faster and has the same output
Off-by-one errors with offsets Offsets are (start, end) in the original string; end is exclusive
Tokenizing with a different tokenizer than the model Always match: use the tokenizer from the same model name
Assuming all tokenizers split the same way BERT uses WordPiece; GPT-2 uses BPE — check with tokenizer.tokenize()

Troubleshooting

Symptom Cause / Fix
ValueError: vocab file not found You passed a vocab path but the file doesn't exist — train first
tokenize() returns [UNK] for everything Vocab is wrong or too small — retrain or check language
encode_batch() crashes on long inputs Enable truncation first
Inconsistent output between slow and fast Shouldn't happen — check tokenizer_class and special tokens
decode returns garbled text (missing spaces) Normalizer stripped whitespace; check ByteLevel pre-tokenizer
Very long sequences get truncated Set max_length in enable_truncation

Official Documentation

Last Updated

2026-09-06 — Initial version.