LLM Hardware¶

LLM hardware decisions are mostly memory decisions. The model has to fit in VRAM or be served in a way that respects latency, concurrency, and cost, and inference uses memory differently than training does.

This cheat sheet focuses on practical LLM hardware considerations: what uses VRAM, what eats it fastest, and how hardware limits shape serving and development.

The Core Constraint¶

For LLMs, VRAM matters because: - Model weights are large. - Inference keeps state around, especially for generation. - Batches and context length multiply memory use. - Quantization and offloading can change the picture dramatically.

A useful way to think about it: - Inference is about fitting the model and serving state while meeting latency/concurrency goals. - Fine-tuning is about fitting weights plus gradients/optimizer state or using a method that avoids full state.

What Uses VRAM During Inference¶

A simple mental model:

Model weights — the biggest fixed cost for a given model and precision.
Activation memory — work memory for each forward pass.
KV cache — cached key/value states for past tokens during generation.
Batch memory — serving multiple sequences at once.
Runtime overhead — framework buffers, temporary tensors, and fragmentation.

For many serving workloads, KV cache and batch size matter as much as the raw model size.

KV Cache¶

During autoregressive generation, the model does not need to recompute every past token's attention from scratch if it can reuse cached key/value states.

That cache grows with: - Context length - Batch size - Number of layers and attention heads - Precision

KV cache is why serving long contexts and large batches can be expensive even when the model weights themselves fit.

Practical implications: - Longer context can cost more inference memory and bandwidth. - Serving many concurrent requests can require a lot of cache. - Some serving strategies optimize cache layout and reuse.

Batch Size and Concurrency¶

Batching helps GPU utilization, but it increases memory use.

Trade-offs: - Larger batches can improve throughput. - Larger batches can increase latency for a given request if scheduling is not careful. - Too large a batch can blow up memory, especially with long contexts. - Some serving systems use continuous or dynamic batching to improve utilization without overcommitting memory.

The "right" batch size depends on the model, hardware, sequence lengths, and whether you care more about throughput or per-request latency.

Precision and Model Size¶

Model size in VRAM depends heavily on precision: - Full precision takes more memory. - FP16/BF16 is common for many models and reduces memory versus FP32. - INT8 and INT4 can reduce memory a lot but change the trade-offs around quality, tooling, and support. - Some models are distributed in multiple formats.

A model that fits comfortably in one precision may not fit in another on the same hardware.

Rule of thumb: - Ask not just "what model?" but "what precision and format?" - Ask "how much context?" and "how many concurrent users?" - Ask "is this inference, distillation, or fine-tuning?"

Quantization¶

Quantization reduces precision, usually to save memory and sometimes to improve throughput.

Common ideas: - Lower-bit weights reduce model memory footprint. - Quantized models can run on smaller GPUs or serve larger models. - Quantization can affect quality, depending on model, calibration, and method. - Inference engines and backends vary in quantization support.

Quantization is not free. It can be: - A great fit for inference on limited hardware - Less attractive for training or certain tasks - Easier or harder depending on the ecosystem

When someone says "this model runs on a consumer GPU," quantization is frequently part of the reason.

Inference Hardware Tiers¶

Very approximate, because real workloads vary widely:

Tier	Typical use cases
Small GPU, a few GB	Small models, prototyping, short context, low concurrency
Consumer-class mid-range GPU	Many smaller LLMs, quantized larger models, dev workflows, local experiments
Higher-memory GPU	Larger models, longer contexts, more concurrency, smoother serving
Multi-GPU or high-memory system	Large models, high concurrency, longer contexts, heavier fine-tuning

The model size, precision, and serving pattern matter more than the marketing name of the GPU.

Fine-Tuning Memory¶

Fine-tuning uses more memory than inference for the same model because of: - Weights - Gradients - Optimizer states - Activations during backward pass

That is why fine-tuning often needs: - More VRAM - Gradient accumulation - Mixed precision - Checkpointing or activation recomputation - Parameter-efficient methods that reduce trainable state

A model that is easy to run for inference may still be hard to fine-tune on the same hardware.

Development vs Serving¶

Development hardware needs: - Enough VRAM to load and run the model - Enough to iterate without constant OOMs - Enough headroom for debugging, logging, and occasional larger runs

Serving hardware needs: - Capacity for expected concurrency and context length - Latency and throughput targets - Stability under load - Room for cache, batching, and overhead

A GPU that is fine for local development may be undersized for production serving, even for the same model.

Practical GPU Selection Questions¶

When choosing hardware for an LLM workload, ask: - What model and precision? - What context length? - How many concurrent requests? - What latency target? - Is this inference, continuous serving, or fine-tuning? - Do we need multi-GPU or is one device enough? - Is quantization acceptable for the quality goals?

If you only ask "which GPU?", you will often overbuy or underbuy.

Simple VRAM Estimation Habits¶

Before sizing hardware, estimate: - Model weights at the chosen precision - KV cache needs for expected context and concurrency - Batch memory - Framework overhead

A model that "just fits" technically may be a poor serving choice if it leaves no room for cache, batching, or multiple users.

Common Hardware Mistakes¶

Assuming model size alone determines fit
Ignoring KV cache and concurrency
Equating inference memory with training memory
Running long-context workloads on hardware sized for short prompts only
Quantizing without checking whether the backend and quality are acceptable
Using a dev box as a production server without sizing for load
Forgetting that Docker, WSL, visibility settings, and driver issues can make hardware look unavailable even when it is present

gpu-basics — VRAM, bandwidth, and why GPUs matter
nvidia-cuda — CUDA stack behind the scenes
cuda-troubleshooting — diagnosing visibility and OOM issues
pytorch-gpu — moving models and tensors to GPU in PyTorch
mlops — deployment, reproducibility, and environment consistency
llm-concepts — quantization and model selection concepts from the LLM angle

Official Documentation¶

NVIDIA GPU and memory docs for the hardware you are considering
Framework docs for device memory behavior and offloading
Serving/runtime docs for batching, cache, and quantization support
Quantization and model release notes for the specific model family you use

Hardware is a constraint, not a guarantee. The model, precision, context, and concurrency usually matter more than the GPU name.

Last Updated¶

2026-09-06 — Initial version.