Hugging Face Datasets¶

The data layer for modern NLP. Hugging Face Datasets gives you one-line access to thousands of pre-built datasets (IMDb, SQuAD, GLUE, etc.), fast map/filter operations, streaming for datasets that don't fit in memory, and integrated evaluation metrics.

Table of Contents¶

Installation / Setup
Core Syntax
Common Operations
Practical Examples
Common Patterns
Best Practices
Common Mistakes
Troubleshooting
Related Cheat Sheets
Official Documentation

Installation / Setup¶

pip install datasets                     # or: uv add datasets
pip install datasets[fsspec]             # for streaming / cloud access

Key concepts:

Concept	What it is
`Dataset`	An immutable table of rows (Arrow format) — like a fast, typed pandas
`DatasetDict`	A dict of splits: `{"train": Dataset, "test": Dataset, ...}`
`load_dataset`	Downloads + caches a dataset from the Hub (or your own files)
`map` / `filter`	Batch operations that work row-by-row or in batches
`features`	Schema: column names and types (`Value("string")`, `ClassLabel`, ...)

Core Syntax¶

Loading datasets¶

from datasets import load_dataset

# from the Hub (by name)
ds = load_dataset("imdb")               # returns DatasetDict with train/test
print(ds)
# DatasetDict({train: Dataset({...}), test: Dataset({...})})

# from your own files
ds = load_dataset("csv", data_files="data.csv")
ds = load_dataset("json", data_files="data.jsonl")
ds = load_dataset("parquet", data_files="data.parquet")

# specific split
train = load_dataset("imdb", split="train")
test = load_dataset("imdb", split="test")

Dataset structure¶

ds = load_dataset("imdb")
print(ds["train"][0])
# {'text': 'I rented I AM CURIOUS-YELLOW...', 'label': 0}

print(ds["train"].column_names)   # ['text', 'label']
print(ds["train"].num_rows)       # 25000
print(ds["train"].features)
# {'text': Value(dtype='string'), 'label': ClassLabel(num_classes=2, names=['neg', 'pos'])}

Map — transform rows¶

def preprocess(batch):
    batch["word_count"] = [len(t.split()) for t in batch["text"]]
    return batch

ds = ds.map(preprocess, batched=True)   # batched=True processes batches, not rows
print(ds["train"][0]["word_count"])      # e.g. 218

batched=True is usually 5-20× faster than per-row for simple operations.

Filter — remove rows¶

ds = ds.filter(lambda x: len(x["text"]) > 50)   # keep long texts
ds = ds.filter(lambda x: x["label"] == 1)       # keep positive only

Select and shuffle¶

ds = ds["train"].shuffle(seed=42).select(range(1000))   # take 1000 random rows

Common Operations¶

Tokenization with the model's tokenizer¶

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, padding=True, max_length=256)

ds = load_dataset("imdb")
ds = ds.map(tokenize, batched=True)
print(ds["train"][0])
# {'text': '...', 'label': 0, 'input_ids': [...], 'attention_mask': [...]}

Setting format for PyTorch / TensorFlow¶

ds.set_format("torch", columns=["input_ids", "attention_mask", "label"])
# returns tensors on access — memory-efficient (doesn't copy)

# or numpy
ds.set_format("numpy", columns=["input_ids", "attention_mask", "label"])

Map with num_proc (parallelism)¶

ds = ds.map(expensive_fn, batched=True, num_proc=4)   # multi-process

From pandas / CSV¶

import pandas as pd
from datasets import Dataset, DatasetDict

df = pd.read_csv("data.csv")
ds = Dataset.from_pandas(df)

# split
split = ds.train_test_split(test_size=0.2, seed=42)
ds = DatasetDict({"train": split["train"], "test": split["test"]})

Practical Examples¶

Example 1 — Full pipeline: load → preprocess → tokenize → train¶

from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                          Trainer, TrainingArguments)
import numpy as np
from sklearn.metrics import accuracy_score

# 1) load
ds = load_dataset("imdb")

# 2) tokenize
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tok(batch):
    return tokenizer(batch["text"], truncation=True, padding=True, max_length=256)

ds = ds.map(tok, batched=True)

# 3) format
ds.set_format("torch", columns=["input_ids", "attention_mask", "label"])

# 4) model
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=2)

# 5) train
args = TrainingArguments("output", num_train_epochs=1, per_device_train_batch_size=32,
                         evaluation_strategy="epoch", fp16=True)
def metrics(eval_pred):
    logits, labels = eval_pred
    return {"accuracy": accuracy_score(labels, np.argmax(logits, axis=1))}

trainer = Trainer(model, args, ds["train"], ds["test"], compute_metrics=metrics)
trainer.train()

Example 2 — Streaming a large dataset¶

# does not download the whole dataset — streams row by row
ds = load_dataset("c4", "en", split="train", streaming=True, trust_remote_code=True)

for i, row in enumerate(ds):
    print(row["text"][:100])
    if i >= 10:
        break

# convert to regular Dataset if needed (limits to sampled rows)
small = ds.take(1000)
small = list(small)   # now in memory

Example 3 — Upload a custom dataset to the Hub¶

from datasets import Dataset

data = {"text": ["Hello", "World"], "label": [0, 1]}
ds = Dataset.from_dict(data)

# push to hub (requires `huggingface_hub login`)
ds.push_to_hub("my-username/my-dataset", private=True)

# or push a DatasetDict
from datasets import DatasetDict
dd = DatasetDict({"train": train_ds, "test": test_ds})
dd.push_to_hub("my-username/my-dataset")

Common Patterns¶

Pattern: text cleaning with map¶

import re

def clean(batch):
    cleaned = []
    for text in batch["text"]:
        text = re.sub(r"<.*?>", " ", text)      # strip HTML
        text = re.sub(r"http\S+", " ", text)     # strip URLs
        text = re.sub(r"\s+", " ", text).strip() # normalize whitespace
        cleaned.append(text)
    batch["text"] = cleaned
    return batch

ds = ds.map(clean, batched=True)

Pattern: class labels¶

ds["train"].features["label"]   # ClassLabel(num_classes=2, names=['neg', 'pos'])
ds["train"].features["label"].int2str(0)   # 'neg'
ds["train"].features["label"].str2int("pos")  # 1

# to add labels from a column
ds = ds.class_encode_column("label")

Pattern: lazy loading for huge data¶

# map returns lazy results until you access them
ds = load_dataset("c4", "en", streaming=True)
ds = ds.map(tokenize_fn, batched=True)    # not materialized yet
ds = ds.shuffle(seed=0).take(1000)       # only 1000 rows materialized

Best Practices¶

Always use batched=True in map — it's dramatically faster and required for batch tokenizers.
Use num_proc for CPU-heavy map functions; avoid it for tokenizers (they have their own parallelism).
For datasets too large for memory: use streaming=True or stream_file=True.
set_format("torch") is lazy — it converts on access, not on call.
Cache processed datasets — map results are cached to disk automatically.
Use DatasetDict with explicit splits (train, validation, test) — not a single Dataset.
Push processed datasets to Hub so you don't reprocess every time.
Always check features to understand column types before mapping.
Use load_dataset("csv", data_files=..., split="train") for local files — split is required for single files.

Common Mistakes¶

Mistake	Fix
`map` without `batched=True` is slow	Always use `batched=True` for most operations
`set_format("torch")` copies the data	No — it's lazy; only converts on access
Confusing `Dataset` with `DatasetDict`	`DatasetDict` has splits; `Dataset` is a single split
Tokenizing row-by-row in `map`	Use `batched=True` + the tokenizer's native batching
Forgetting `num_proc` for heavy transforms	`num_proc=4` speeds up CPU-bound `map` operations
`streaming=True` + `shuffle()` gives poor randomness	`shuffle(seed, buffer_size=1000)` for approximate streaming shuffle
`push_to_hub` fails with auth error	Run `huggingface-cli login` first
Dataset has wrong label encoding	Check `features["label"]` and use `class_encode_column`

Troubleshooting¶

Symptom	Cause / Fix
`FileNotFoundError` on `load_dataset`	Check the dataset name; sometimes requires `trust_remote_code=True`
`ArrowInvalid: offset too large`	Dataset is Arrow-backed; convert to pandas for complex indexing
`map` returns inconsistent column counts	Ensure the function returns the same keys for every batch
Memory blow-up during `map`	Use `batched=True` + `num_proc`, or use streaming
`set_format` errors on column	Column doesn't exist; check `column_names`
`tokenizer` returns wrong shapes	Use `return_tensors=None` (not `"pt"`) inside `map` for HF datasets
Slow first load	Datasets are cached to `~/.cache/huggingface/datasets/` — first run downloads
`FileNotFoundError: data/...` on cache	Clear cache: `rm -rf ~/.cache/huggingface/datasets/<name>`

huggingface-transformers — models and fine-tuning
tokenizers — fast subword tokenization
sentence-transformers — semantic embeddings
nlp-preprocessing — preprocessing pipelines
pandas — DataFrame operations
ML evaluation — model evaluation
Environment variables — HF tokens and secrets

Official Documentation¶

HF Datasets docs — https://huggingface.co/docs/datasets
HF Hub datasets — https://huggingface.co/datasets
Datasets Quickstart — https://huggingface.co/docs/datasets/quickstart

Last Updated¶

2026-09-06 — Initial version.

Hugging Face Datasets¶

Table of Contents¶

Installation / Setup¶

Core Syntax¶

Loading datasets¶

Dataset structure¶

Map — transform rows¶

Filter — remove rows¶

Select and shuffle¶

Common Operations¶

Tokenization with the model's tokenizer¶

Setting format for PyTorch / TensorFlow¶

Map with num_proc (parallelism)¶

From pandas / CSV¶

Practical Examples¶

Example 1 — Full pipeline: load → preprocess → tokenize → train¶

Example 2 — Streaming a large dataset¶

Example 3 — Upload a custom dataset to the Hub¶

Common Patterns¶

Pattern: text cleaning with map¶

Pattern: class labels¶

Pattern: lazy loading for huge data¶

Best Practices¶

Common Mistakes¶

Troubleshooting¶

Related Cheat Sheets¶

Official Documentation¶

Last Updated¶