Hugging Face Transformers

The library that made pre-trained NLP models accessible to everyone. Transformers gives you one-line pipelines for classification, summarization, question-answering, and more — plus the full Trainer API for fine-tuning BERT, GPT, T5, and hundreds of other models on your own data.

Table of Contents

• Installation / Setup
• Core Syntax
• Common Operations
• Practical Examples
• Common Patterns
• Best Practices
• Common Mistakes
• Troubleshooting
• Related Cheat Sheets
• Official Documentation

Installation / Setup

pip install transformers torch       # or: uv add transformers torch
# CPU-only:
pip install transformers[torch]      # or: uv add transformers torch --extra-index-url https://download.pytorch.org/whl/cpu

Key packages:

Package	Role
transformers	Models, tokenizers, pipelines, Trainer
torch / tensorflow	Backend (PyTorch or TF)
datasets	Hugging Face datasets library
accelerate	Multi-GPU / mixed-precision for Trainer

Core Syntax

Pipeline — zero-code inference

from transformers import pipeline

# text classification (sentiment)
classifier = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
print(classifier("This movie was amazing!"))
# [{'label': 'POSITIVE', 'score': 0.9998}]

# summarization
summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
print(summarizer("Long text here...", max_length=100))

# question-answering
qa = pipeline("question-answering")
print(qa(question="What is Hugging Face?", context="Hugging Face builds tools for ML."))

# fill-mask
fill = pipeline("fill-mask", model="bert-base-uncased")
print(fill("The capital of France is [MASK]."))

# zero-shot classification
zero = pipeline("zero-shot-classification")
print(zero("I love playing guitar", candidate_labels=["music", "sports", "cooking"]))

Task	Pipeline name	Model suggestion
Sentiment	text-classification	distilbert-base-uncased-finetuned-sst-2-english
Summarization	summarization	facebook/bart-large-cnn, t5-small
Question-answering	question-answering	deepset/roberta-base-squad2
Fill mask	fill-mask	bert-base-uncased
Zero-shot	zero-shot-classification	facebook/bart-large-mnli
NER	token-classification	dbmdz/bert-large-cased-finetuned-conll03-english
Text generation	text-generation	gpt2, meta-llama/Llama-3-8B-Instruct
Translation	translation	t5-small, Helsinki-NLP/opus-mt-en-de
Feature extraction	feature-extraction	sentence-transformers/all-MiniLM-L6-v2

AutoModel / AutoTokenizer — load any pre-trained model

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

# tokenize and forward pass
inputs = tokenizer("Hello world", return_tensors="pt", truncation=True, padding=True)
outputs = model(**inputs)
print(outputs.logits)      # raw logits for 2 classes

Fine-tuning with Trainer

from transformers import (AutoModelForSequenceClassification, AutoTokenizer,
                          Trainer, TrainingArguments)
from datasets import load_dataset
import numpy as np
from sklearn.metrics import accuracy_score

# 1) data
ds = load_dataset("imdb")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, padding=True, max_length=256)

ds = ds.map(tokenize, batched=True)
ds = ds.rename_column("label", "labels")
ds.set_format("torch", columns=["input_ids", "attention_mask", "labels"])

# 2) model + tokenizer
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=2)

# 3) metrics
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=1)
    return {"accuracy": accuracy_score(labels, preds)}

# 4) training args
args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=2,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    eval_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    weight_decay=0.01,
    load_best_model_at_end=True,
    fp16=True,                    # mixed precision
)

# 5) train
trainer = Trainer(
    model=model, args=args,
    train_dataset=ds["train"], eval_dataset=ds["test"],
    compute_metrics=compute_metrics,
)
trainer.train()
print(trainer.evaluate())
trainer.save_model("./best_model")

Common Operations

Tokenization details

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokens = tokenizer("Hello world!", return_tensors="pt", truncation=True, max_length=128)
# {'input_ids': tensor([[101, 7592, 2088, 999, 102]]),
#  'attention_mask': tensor([[1, 1, 1, 1, 1]])}

# decode back
tokenizer.decode([101, 7592, 2088, 999, 102])
# '[CLS] hello world! [SEP]'

# batch tokenization (much faster)
tokenizer(["Hello", "Goodbye"], padding=True, truncation=True, return_tensors="pt")

Argument	Purpose
return_tensors="pt" / "tf" / "np"	Output format (PyTorch / TensorFlow / NumPy)
truncation=True	Truncate to max_length
padding=True or "max_length"	Pad shorter sequences
max_length=128	Maximum sequence length
return_attention_mask=True	Include attention mask (default)
return_offsets_mapping=True	Character offsets for alignment (NER)

Saving and loading

# save
model.save_pretrained("./my_model")
tokenizer.save_pretrained("./my_model")

# load
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained("./my_model")
tokenizer = AutoTokenizer.from_pretrained("./my_model")

# from a hub model
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased")

Model families

Family	Architecture	Best for
BERT	Bidirectional encoder	Classification, NER, QA
RoBERTa	BERT variant (better pretraining)	Same as BERT, often better
GPT-2 / GPT-3	Autoregressive decoder	Text generation
T5 / FLAN-T5	Encoder-decoder	Translation, summarization, any seq-to-seq
DistilBERT	Smaller, faster BERT	When speed matters
LLaMA	Open autoregressive decoder	Instruction following, generation
ALBERT	Parameter-sharing BERT	Same tasks, smaller model

Practical Examples

Example 1 — Zero-shot text classification

from transformers import pipeline

zero = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
texts = [
    "The stock market rallied 5% today on strong earnings",
    "Scientists discovered a new exoplanet orbiting a nearby star",
    "The government announced new climate change legislation",
]
labels = ["finance", "science", "politics"]

for text in texts:
    result = zero(text, candidate_labels=labels, multi_label=False)
    print(f"{result['labels'][0]:10s} ({result['scores'][0]:.2f}): {text[:60]}")

Example 2 — Fine-tune BERT for custom NER

from transformers import (AutoModelForTokenClassification, AutoTokenizer,
                          Trainer, TrainingArguments, DataCollatorForTokenClassification)
from datasets import load_dataset

ds = load_dataset("conll2003")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

def tokenize_labels(batch):
    tokens = tokenizer(batch["tokens"], truncation=True, is_split_into_words=True, max_length=128)
    labels = []
    for i, label in enumerate(batch["ner_tags"]):
        word_ids = tokens.word_ids(batch_index=i)
        label_ids = []
        prev = None
        for word_id in word_ids:
            if word_id is None:
                label_ids.append(-100)
            elif word_id != prev:
                label_ids.append(label[word_id])
            else:
                label_ids.append(-100)   # subword
            prev = word_id
        labels.append(label_ids)
    tokens["labels"] = labels
    return tokens

ds = ds.map(tokenize_labels, batched=True)
model = AutoModelForTokenClassification.from_pretrained("bert-base-cased", num_labels=9)
data_collator = DataCollatorForTokenClassification(tokenizer)

trainer = Trainer(
    model=model,
    args=TrainingArguments(output_dir="./ner_results", num_train_epochs=3,
                          per_device_train_batch_size=16, per_device_eval_batch_size=32),
    train_dataset=ds["train"], eval_dataset=ds["validation"],
    data_collator=data_collator,
)
trainer.train()

Example 3 — Use a T5 model for text-to-text tasks

from transformers import pipeline

t5 = pipeline("text2text-generation", model="t5-small")

# summarization
summary = t5("summarize: " + long_text, max_length=100)

# translation
translation = t5("translate English to French: Hello, how are you?")

# classification (T5 treats all tasks as text-to-text)
label_text = t5("mnli premise: I am hungry. hypothesis: I want food.")[0]["generated_text"]
print(label_text)  # "entailment"

Common Patterns

Pattern: semantic search with feature extraction

from transformers import pipeline
import numpy as np

fe = pipeline("feature-extraction", model="sentence-transformers/all-MiniLM-L6-v2")

docs = ["Python is great for data science", "JavaScript runs in the browser", "R is popular for statistics"]
doc_vecs = np.array([np.mean(fe(d)[0], axis=0) for d in docs])

query_vec = np.mean(fe("data analysis tools")[0], axis=0)
sims = np.dot(doc_vecs, query_vec) / (np.linalg.norm(doc_vecs, axis=1) * np.linalg.norm(query_vec))
print(docs[np.argmax(sims)])
# "Python is great for data science"

Pattern: batch inference with pipe

from transformers import pipeline

classifier = pipeline("text-classification", device=0)   # GPU

texts = ["Great product", "Terrible service", "Absolutely fantastic"]
# pipe handles batching automatically
results = classifier(texts, batch_size=32)

Pattern: custom model head

from transformers import AutoModel
import torch

model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
backbone = AutoModel.from_pretrained(model_name)

inputs = tokenizer("Embed this text", return_tensors="pt")
outputs = backbone(**outputs)
# outputs.last_hidden_state: (batch, seq_len, hidden_dim) — e.g. [CLS] embedding
cls_embedding = outputs.last_hidden_state[:, 0, :]   # CLS token

Best Practices

• Use pipeline for quick experiments — it handles tokenization + decoding + batching automatically.
• For fine-tuning: Trainer + TrainingArguments is the standard workflow; set fp16=True for GPU speed.
• Always set truncation=True and a reasonable max_length — models have fixed maximum input lengths.
• Use device=0 to move inference to GPU; device=-1 for CPU.
• model.to(device) before manual forward passes (Trainer handles this automatically).
• AutoModelForSequenceClassification / AutoModelForTokenClassification add task-specific heads — not AutoModel.
• Save both model and tokenizer together — the tokenizer is required for decoding.
• Use datasets for loading training data (streaming for large datasets).

Common Mistakes

Mistake	Fix
CUDA out of memory	Lower per_device_train_batch_size, enable fp16, or use gradient accumulation
tokenizer gives unexpected tokens	Check the model's tokenizer type: BERT uses [CLS], GPT-2 uses Ġ
Model outputs logits, not probabilities	Use torch.nn.functional.softmax() on outputs.logits
Confusion between AutoModel and AutoModelForSequenceClassification	The For... variants add task heads; AutoModel is just the backbone
Training too slow	Enable fp16=True, increase batch size, reduce max_length
KeyError: 'labels' during evaluation	You didn't provide labels — check set_format and rename_column
Model too large for CPU inference	Use pipeline which auto-downloads small models; or use device_map="auto"
Missing trust_remote_code=True for custom models	Required when loading community models with custom code

Troubleshooting

Symptom	Cause / Fix
OSError: model not found	Check the model name on hub; use full name bert-base-uncased
RuntimeError: CUDA out of memory	Lower batch size, enable fp16, use gradient_checkpointing=True
ValueError: No tokenizer found	You loaded AutoModel but need the tokenizer — call AutoTokenizer.from_pretrained
Strange tokenization output	Some models use SentencePiece (▁) or BPE (Ġ) — this is expected
model.generate() returns garbage	Use model.eval() + torch.no_grad() during inference
Fine-tuned model worse than baseline	Too few epochs, wrong learning rate, or data too small
ImportError: cannot import name 'Trainer'	Install accelerate and datasets alongside transformers
Slow inference on CPU	Use distilbert or smaller models; GPU for production

Related Cheat Sheets

• huggingface-datasets — loading and processing training data
• tokenizers — fast subword tokenization
• sentence-transformers — semantic embeddings
• spacy — fast production NLP
• nlp-preprocessing — when to preprocess vs raw input
• ML evaluation — evaluating fine-tuned models
• Environment variables — Hugging Face tokens and secrets

Official Documentation

• Transformers docs — https://huggingface.co/docs/transformers
• Transformers tutorial — https://huggingface.co/docs/transformers/training
• Hugging Face Hub — https://huggingface.co/models

Last Updated

2026-09-06 — Initial version.