Skip to content

Python Standard Library

The batteries included with Python that data scientists reach for daily: files, data formats, high-performance collections, and functional tools. Zero installs required.

Table of Contents

Installation / Setup

All modules below ship with Python. import them directly — nothing to install.

import json, csv, re, math
from pathlib import Path
from collections import Counter, defaultdict, deque, namedtuple
import itertools, functools

Core Syntax

Paths: pathlib.Path

from pathlib import Path

p = Path("data/raw/train.csv")
p.name            # 'train.csv'
p.stem            # 'train'
p.suffix          # '.csv'
p.parent          # PosixPath('data/raw')
p.exists()        # False
Path("out").mkdir(parents=True, exist_ok=True)

text = Path("notes.txt").read_text(encoding="utf-8")
Path("out.json").write_text(json.dumps(data), encoding="utf-8")

for f in Path("data").glob("*.csv"):     # glob: pattern matching
    ...
for f in Path("data").rglob("*.parquet"):  # recursive
    ...

JSON

import json

s = json.dumps({"a": 1}, indent=2)      # dict -> str
data = json.loads(s)                    # str -> dict
data = json.load(open("cfg.json"))      # from file
json.dump(data, open("out.json", "w"), indent=2)

CSV

import csv

with open("data.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)          # rows as dicts using header
    rows = list(reader)

with open("out.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "score"])
    writer.writeheader()
    writer.writerows(rows)

collections

Type Use case
defaultdict(factory) Dict that auto-creates missing keys
Counter(iterable) Count occurrences; .most_common(n)
namedtuple("Name", "a b") Lightweight record (immutable)
deque(maxlen=n) Fast append/pop from both ends; ring buffer
OrderedDict Remembers insertion order (dicts do too since 3.7) 
from collections import Counter, defaultdict, namedtuple

Counter(["a", "b", "a"]).most_common(1)          # [('a', 2)]
counts = defaultdict(int); counts["x"] += 1       # no KeyError
Point = namedtuple("Point", "x y"); Point(1, 2).x  # 1

itertools

Function Result
product(a, b) Cartesian product (grid search!)
permutations(xs, n) / combinations(xs, n) Orderings / subsets
chain(a, b) Iterate over several iterables as one
groupby(xs, key) Group adjacent items by key (sort first!)
islice(it, n) First n items (lazy slice)
count(start, step) Infinite counter
cycle(xs) Repeat forever (e.g., epoch cycling)
repeat(x, n) Same value n times 
import itertools as it

list(it.product([0.1, 0.01], [100, 200]))     # hyperparameter grid
[(k, list(g)) for k, g in it.groupby(sorted(xs), key=lambda x: x["cat"])]

functools

import functools

functools.partial(round, ndigits=2)   # pre-fill arguments
@functools.lru_cache(maxsize=128)     # memoize pure functions
functools.reduce(operator.add, [1, 2, 3])  # fold

Math & statistics

import math, statistics

math.sqrt(16); math.log(100, 10); math.inf; math.nan
statistics.mean(xs); statistics.median(xs); statistics.stdev(xs)

Logging

import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)
logger.info("training started, lr=%s", lr)   # lazy formatting

Use logging instead of print for anything that runs unattended (training scripts, pipelines).

Common Operations

Sorting and grouping

sorted(rows, key=lambda r: r["date"])              # stable sort
max(rows, key=lambda r: r["score"])                # argmax
sorted(set(all_tags))                              # unique sorted

Command-line arguments

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--lr", type=float, default=0.01)
parser.add_argument("--data", required=True)
args = parser.parse_args()                          # args.lr, args.data

Shuffle / sample (reproducible)

import random
random.seed(42)
random.shuffle(xs)
sample = random.sample(xs, k=5)

SQLite (zero-config database)

import sqlite3

con = sqlite3.connect("mydb.sqlite")
con.execute("CREATE TABLE IF NOT EXISTS t (x REAL)")
con.executemany("INSERT INTO t VALUES (?)", [(1.0,), (2.0,)])
rows = con.execute("SELECT * FROM t").fetchall()
con.commit()

Subprocess (run shell commands)

import subprocess
out = subprocess.run(["git", "status"], capture_output=True, text=True)
print(out.stdout)

Copy

import copy
b = copy.deepcopy(a)      # fully independent copy (lists, dicts, objects)

Practical Examples

1. Read config, load data, log progress

import json, logging
from pathlib import Path

logging.basicConfig(level=logging.INFO)
cfg = json.loads(Path("config.json").read_text(encoding="utf-8"))

for file in Path("data").glob("*.json"):
    logging.info("processing %s", file.name)
    data = json.loads(file.read_text(encoding="utf-8"))

2. Class distribution with Counter

from collections import Counter

labels = ["cat", "dog", "cat", "bird", "cat", "dog"]
Counter(labels).most_common()          # [('cat', 3), ('dog', 2), ('bird', 1)]

3. Chunk a list lazily

import itertools as it

def chunks(xs, n):
    itr = iter(xs)
    return iter(lambda: list(it.islice(itr, n)), [])

for batch in chunks(range(1000), 100):
    train(batch)

Common Patterns

# defaultdict for grouping
from collections import defaultdict
by_city = defaultdict(list)
for row in rows:
    by_city[row["city"]].append(row)

# deque as a rolling window
from collections import deque
window = deque(maxlen=5)          # auto-drops oldest
for x in stream: window.append(x)

# pathlib for reproducible paths
BASE = Path(__file__).resolve().parent
data_path = BASE / "data" / "train.csv"

# namedtuple for typed-ish rows
Record = namedtuple("Record", "id value")
recs = [Record(r["id"], r["value"]) for r in rows]

Best Practices

  • Use pathlib for all path work — no string concatenation of paths.
  • Use Counter/defaultdict instead of manual counting/grouping.
  • Sort before groupby — it only groups adjacent equal keys.
  • Prefer itertools over manual loop counters (enumerate, zip, product).
  • Use logging, not print, in scripts and training loops.
  • Always pass encoding="utf-8" when opening text files.
  • Use copy.deepcopy deliberately — it can be slow on large structures.
  • Use random.seed(42) for reproducible sampling and shuffles.

Common Mistakes

Mistake Consequence Correct approach
groupby without sorting Splits same-key groups apart sorted(xs, key=k) first
Path.glob("*") missing nested files Empty results rglob or explicit pattern
Reading huge CSV with read() Memory blowup Iterate the reader object
copy.copy on nested structures Shared inner objects copy.deepcopy
print in production scripts No timestamps/levels, lost in CI logging

Troubleshooting

Symptom Cause Fix
json.JSONDecodeError Malformed file / wrong encoding encoding="utf-8"; validate with a JSON linter
PermissionError on write Path is a directory or read-only Check the target path
StopIteration in islice/next Iterator exhausted Provide defaults or catch
TypeError: expected str, got Path Path passed where str needed str(p) (or use APIs that accept Path)
sqlite3.OperationalError: no such table Table created in different DB file Check sqlite3.connect path

Official Documentation

Last Updated

2026-09-06 — Initial version.