NVIDIA CUDA¶
CUDA is NVIDIA's parallel computing platform and programming model. For most AI practitioners, the important part is not writing kernels by hand, but understanding the layers below the frameworks and what can go wrong.
CUDA is a stack, not a single thing. It includes: - A programming model for GPU compute - Compilers and toolchain - Libraries for common math and primitives - A driver/runtime layer that talks to the hardware - Profiling and debugging tools
Frameworks like PyTorch and TensorFlow use CUDA under the hood. You can also use CUDA directly from C/C++, and indirectly from higher-level languages through bindings.
The CUDA Stack in One Picture¶
- Application code — your model, your kernel, or your framework call
- Libraries — cuBLAS, cuDNN, cuFFT, NCCL, and many others
- CUDA runtime — the API your code uses to launch work on the GPU
- Driver — the system-level piece that manages the GPU and exposes it to the OS and runtime
- Hardware — the GPU itself
You usually touch the runtime and libraries. The driver is what makes the hardware accessible. When something "CUDA-related" breaks, the issue can be anywhere in this stack.
Toolkit vs Driver¶
A common source of confusion is mixing up: - CUDA toolkit — compiler, libraries, headers, developer tools - CUDA driver — the NVIDIA driver that supports CUDA at the system level
The toolkit version and driver version are related but not identical. A newer toolkit can often work with an older driver up to a point, and mismatches can cause runtime errors.
Practical rules: - Keep the driver compatible with the toolkit and frameworks you use. - If you install a framework that bundles CUDA, use the CUDA version it expects. - If you install CUDA separately, match it to your framework's expectations. - Verify both driver and toolkit when diagnosing problems.
CUDA Runtime Basics¶
The CUDA runtime API lets code: - Allocate and free device memory - Copy data between host and device - Launch kernels - Synchronize - Use streams and events - Query device properties
These operations map to the GPU as a separate device with its own memory and execution model. That separation is why moving data to and from the GPU matters so much.
Device Memory¶
CUDA device memory is separate from CPU memory. You have to: - Allocate on the device - Copy input data to the device - Launch work - Copy results back if needed
Two kinds of copies often matter: - Host to device — upload data to the GPU - Device to host — download results
Implicit copies happen in frameworks too. PyTorch tensors with a CUDA device live in GPU memory; moving them between CPU and GPU is an explicit or framework-mediated transfer.
Streams and Concurrency¶
CUDA streams let work execute concurrently or in order, depending on how you use them.
- Default stream — the common implicit stream
- Explicit streams — allow overlapping copies and compute, or ordering work deliberately
Streams matter when you want to: - Overlapped data transfer and computation - Control the order of operations - Improve utilization by keeping more of the GPU busy
You do not always need explicit streams in Python-frameworks usage, but the concept explains why some pipelines overlap transfers and compute while others do not.
Kernels¶
A CUDA kernel is a function that runs on the GPU, launched by many threads in parallel.
Conceptually: - You write the kernel - You specify how many threads and blocks to use - The GPU executes it across the device
Most AI practitioners do not write kernels daily, but libraries like cuDNN and framework backends do. Understanding kernels helps when you read profiling output or write custom operations.
Libraries You Will Likely Encounter¶
| Library | What it helps with |
|---|---|
| cuBLAS | Dense linear algebra, matrix multiplies |
| cuDNN | Deep learning primitives, convolutions, activations |
| cuFFT | Fast Fourier transforms |
| NCCL | Multi-GPU communication primitives |
| CUDA runtime libs | Core runtime support |
Frameworks usually use these internally. When people say "CUDA performance," part of what they mean is good use of these libraries and the underlying hardware.
Compilation and PTX¶
CUDA code is compiled to GPU-executable form. The toolchain can emit intermediate representations such as PTX, which is a device assembly-like intermediate language.
You may see PTX or compute capability references when: - Installing compiled CUDA packages - Reading compatibility messages - Debugging driver/toolkit mismatches
It is enough to know that CUDA code is not portable across every GPU generation automatically; the toolchain targets specific compute capabilities and architectures.
Compute Capability¶
Compute capability is NVIDIA's way of identifying GPU architecture generations and features.
Why it matters: - Some features require a minimum compute capability - Precompiled binaries may target a specific architecture - Compatibility issues often show up as compute capability or architecture messages
If you are not writing or compiling CUDA directly, you mostly encounter this when installing packages or diagnosing hardware support questions.
CUDA from Python¶
Most AI practitioners touch CUDA through a framework, not raw CUDA C++.
Common Python-facing concepts: - Selecting a device - Moving tensors to CUDA - Synchronizing when needed - Checking availability - Seeing GPU memory usage - Profiling GPU work
Frameworks hide much of the runtime, but the underlying model is still there: device memory, copies, launches, and synchronization.
CUDA and the Rest of the System¶
CUDA does not live in isolation: - The OS must see the GPU through the driver - Multiple processes can contend for GPUs - Visible devices can be controlled by environment variables or APIs - Container runtimes need access to the GPU and driver libraries - Docker and similar tools often need explicit GPU passthrough
If PyTorch cannot see a GPU, the cause can be driver, toolkit, framework build, container config, or device visibility rather than the code itself.
Common CUDA Signals¶
| Symptom | Likely layer |
|---|---|
| "CUDA not available" | driver, toolkit, framework build, device visibility, container access |
| "Device / device memory" errors | allocation, out-of-memory, misuse of device tensors |
| Slow transfers | host-device copies, PCIe limits, missing overlap |
| Kernel launch failures | mismatched execution parameters or illegal configuration |
| Version mismatch errors | driver/toolkit/framework disagreement |
When something CUDA-related breaks, think in layers: hardware, driver, toolkit, libraries, framework, and then your code.
Practical Checkpoints¶
- Does the system see the GPU?
- Is the driver present and compatible?
- Does the framework see the GPU?
- Can you allocate and compute on it?
- Are you moving data efficiently?
- Are you running out of device memory?
- Is the GPU actually busy, or waiting on the host?
The sibling cheat sheet cuda-troubleshooting goes deeper on diagnosing these.
Related Cheat Sheets¶
- gpu-basics — hardware concepts behind CUDA
- cuda-troubleshooting — diagnosing CUDA and GPU problems
- pytorch-gpu — using CUDA from PyTorch
- llm-hardware — CUDA in the context of LLM workloads
- linux-commands — the system commands that often help diagnose GPU/CUDA issues
- mlops — CUDA consistency matters a lot in reproducible environments
Official Documentation¶
- NVIDIA CUDA programming guide and toolkit documentation
- NVIDIA driver documentation and compatibility notes
- cuBLAS, cuDNN, NCCL library documentation as relevant
- Your framework's CUDA/device documentation
CUDA is large. For day-to-day AI work, focus on availability, device memory, transfers, streams/concurrency, and the driver-toolkit relationship.
Last Updated¶
2026-09-06 — Initial version.