GPU Basics¶

GPUs are not just "faster CPUs." They are throughput-oriented parallel processors optimized for massive, regular computation — which is exactly what matrix math in ML and AI looks like.

This cheat sheet is the hardware base layer. CUDA programming and PyTorch GPU workflows live in sibling sheets, but you need the concepts here first.

CPU vs GPU — The Right Mental Model¶

Property	CPU	GPU
Design goal	Low-latency, general-purpose tasks	High-throughput, parallel math
Core count	Few to dozens of complex cores	Thousands of simpler cores
Strength	Branching, serial logic, I/O	Dense matrix/tensor operations
Memory	Smaller, lower latency	Large, higher bandwidth
Best for	Control flow, data loading, preprocessing	Training, inference, big linear algebra

A GPU wins when work is: - Parallel — thousands of independent or nearly-independent operations - Dense — matrix multiplies, convolutions, vector math - Regular — predictable memory access patterns

A GPU loses when work is: - Serial — one step depends heavily on the previous one - Branch-heavy — lots of divergent control flow - Small — overhead dominates the actual math

What's Actually on a GPU¶

A modern GPU contains:

Streaming Multiprocessors (SMs) — the main compute blocks
cuda cores / shader cores / stream processors — the individual arithmetic units inside SMs
Tensor cores (on many newer chips) — specialized units for matrix math, especially useful for quantized and mixed-precision work
Memory — VRAM, usually GDDR or HBM, separate from system RAM
Memory controllers and buses — determine how fast data gets in and out
Interconnects — PCIe, NVLink, or similar for talking to the CPU and other GPUs

The name "CUDA core" is NVIDIA-specific, but the idea is generic: a GPU is a big collection of simple parallel execution units.

VRAM — Why It Matters So Much¶

VRAM is the GPU's working memory. It holds: - Model weights - Activations during training - Activations and KV cache during inference - Optimizer states during training - Intermediate tensors, gradients, and temporary buffers

VRAM is often the bottleneck before compute is. You can have a fast GPU and still be unable to train a model if it does not fit.

Rough rule of thumb: - Larger models need more VRAM. - Larger batch sizes need more VRAM. - Training needs far more VRAM than inference for the same model, because of optimizer states and gradients. - Mixed precision and quantization reduce VRAM pressure.

VRAM capacity tiers in practice¶

Tier	Typical use
Small (a few GB)	Inference for modest models, learning, small experiments
Medium (8–24 GB)	Fine-tuning smaller models, inference for many LLMs and vision models
Large (40–80+ GB)	Bigger fine-tuning, larger batches, multi-model work, heavier inference
Multi-GPU / HBM systems	Very large training and inference workloads

The actual number depends on model size, batch size, precision, and framework overhead. Always think in terms of what you are trying to fit, not just the raw GPU spec.

Memory Bandwidth¶

Memory bandwidth is how much data the GPU can read/write per second. It often matters more than raw compute for many workloads.

Why? - Model weights and activations must move through memory. - If the GPU spends a lot of time waiting for data, the compute units sit idle. - Bandwidth-limited workloads do not benefit much from more compute.

This is one reason large models and large batches are sensitive to both VRAM capacity and bandwidth.

CUDA Cores and Parallelism¶

CUDA cores execute the fine-grained math operations. A kernel launch launches many threads that run in parallel across these cores.

Key ideas: - Thousands of threads can run simultaneously. - Threads are grouped into blocks and grids. - The GPU scheduler tries to keep the pipelines full. - If your work is not parallel enough, much of the hardware sits unused.

You do not need to write CUDA kernels to use a GPU effectively, but understanding that GPUs are massive parallelism engines helps explain why some workloads fly and others stall.

Compute Precision¶

GPUs support different numeric precisions: - FP32 — full precision, higher memory and compute cost - FP16 / half precision — lower memory, often faster, but not always stable - BF16 — a alternative low-precision format with different trade-offs from FP16 - INT8 / INT4 — heavily quantized formats for inference and some training tricks - TF32 — on some NVIDIA hardware, a Tensor-core-friendly format that is faster than FP32 but not identical in result

Mixed precision training uses lower precision for much of the compute while keeping a master copy or select quantities in higher precision. The practical result is often faster training and lower memory use, with care.

GPU Memory Hierarchy¶

Like CPUs, GPUs have a memory hierarchy: - Registers — per-thread fast storage - Shared memory — fast on-chip memory visible to threads in a block - L2 / cache — larger, slower on-chip storage - VRAM — the large device memory

The exact details matter most for kernel writers. For practitioners, the big picture is enough: data locality and memory movement cost matter a lot.

When GPU Choose Matter¶

A GPU is a good fit when: - You are doing large matrix/tensor math. - Your model or data is big enough to keep the device busy. - You can feed data fast enough not to starve the GPU. - You can fit the model and working state in VRAM or a reasonable sharding/offload strategy.

A GPU is a poor fit when: - The problem is tiny. - The workload is mostly serial or I/O-bound. - You are limited by host-side preprocessing or data loading more than by the GPU. - VRAM is too small for the model and strategy you want.

Practical Indicators You Are GPU-Bound or Memory-Bound¶

GPU-bound — compute is the limit; utilization is high; adding more math or a bigger batch may help if memory allows.
Memory-bound — the GPU waits on data movement; bandwidth and capacity matter more than raw FLOPs.
I/O or CPU-bound — the GPU waits for the host; fix data loading, shuffling, prefetching, or preprocessing first.

In PyTorch, torch.cuda.utilization and profiling tools help you tell these apart. In practice, many projects start CPU- or I/O-bound and become GPU-bound only after the data pipeline is solid.

Multi-GPU Concepts¶

When one GPU is not enough:

Data parallelism — same model on multiple devices, split the batch
Model parallelism — split the model across devices
Pipeline parallelism — split layers into stages across devices
Distributed training — coordinate many devices with a framework

Multi-GPU work introduces communication overhead. It helps when the computation per device is large relative to the communication cost.

nvidia-cuda — CUDA ecosystem, runtime, toolkit, driver model
cuda-troubleshooting — diagnosing GPU and CUDA problems
pytorch-gpu — PyTorch device management, data movement, and GPU checks
llm-hardware — hardware considerations for LLM inference and fine-tuning
python-testing — test GPU code paths carefully when you can
mlops — hardware strategy is part of deployment and reproducibility

Official Documentation¶

There is no single universal GPU manual; consult: - NVIDIA GPU and CUDA documentation for NVIDIA hardware - Your hardware vendor's architecture and memory docs - The framework docs for device behavior (PyTorch, TensorFlow, JAX, etc.)

The concepts here are hardware-agnostic; the specifics are vendor and framework specific.

Last Updated¶

2026-09-06 — Initial version.