CUDA Troubleshooting¶

CUDA problems usually look like one of a few things: the GPU is not visible, the driver and toolkit disagree, the device runs out of memory, or the code is spending its time moving data instead of computing. Start by locating the layer that is failing.

This is a defensive diagnostic cheat sheet. It focuses on finding the cause, not on hacking around it.

First Question: Does the System See the GPU?¶

Before blaming CUDA or the framework, check the hardware/driver layer.

nvidia-smi¶

nvidia-smi

Look for: - The GPU listed at all - Driver version - GPU utilization - Memory usage - Temperature and power, when relevant - Any processes using the GPU

If nvidia-smi does not show the GPU, the problem is usually driver, hardware, or system configuration.

Quick liveness checks¶

nvidia-smi -L
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv

These answer: how many GPUs are visible, and what driver is present?

If the GPU is missing here, CUDA availability from a framework is unlikely to work.

CUDA Availability Checklist¶

When PyTorch, TensorFlow, JAX, or a similar framework says CUDA is unavailable, walk the layers:

Hardware present? — nvidia-smi shows a GPU.
Driver present and loaded? — nvidia-smi shows a driver version.
Toolkit/libraries match? — the framework or CUDA install matches the driver.
Framework built with CUDA? — the framework binary supports CUDA.
Visible devices configured? — CUDA_VISIBLE_DEVICES or container/device mapping may hide GPUs.
Container/cgroup access? — Docker, WSL, or similar may need GPU passthrough.
Permissions/OS issues? — rare, but possible.

Most "CUDA not available" issues are one of the first five.

Driver vs Toolkit Mismatch¶

A classic failure mode: - The toolkit expects a newer driver than the system has - Or the installed driver is newer than the toolkit expects, which can also cause odd behavior - Or a framework bundles its own CUDA and conflicts with a system CUDA

Symptoms: - Runtime errors after a successful import - "cudaError" style messages - Unexpected "unsupported" messages - Failure only when a CUDA operation is actually attempted

Diagnostic questions: - What does nvidia-smi report? - What CUDA version does the framework report? - Did you install CUDA separately, or use a framework that bundles it? - Are you mixing a system CUDA install with a bundled-framework CUDA?

Practical rule¶

If a framework bundles CUDA, prefer using that bundled toolchain rather than mixing multiple CUDA installs unless you know why you are doing it.

Out of Memory¶

CUDA out-of-memory is one of the most common GPU errors. It usually means the working set on the GPU is too large.

Common contributors: - Model weights - Optimizer states - Gradients - Activations - Batch size too large - Temporary buffers and fragmentation - Multiple processes sharing the GPU

Diagnostic steps¶

Check current memory use in nvidia-smi while the job runs.
Print or log device memory stats from the framework if available.
Reduce batch size.
Reduce precision where appropriate.
Use gradient accumulation if you need a larger effective batch but cannot fit it.
Avoid keeping unnecessary CUDA tensors alive.
Check for hidden copies or duplicated tensors.

Memory debugging mindset¶

Before assuming the model is just too big, check: - Are you moving data to CUDA more than needed? - Are you storing intermediate tensors you do not need? - Is there a loop or dataloader issue creating extra work? - Is the framework caching memory in a way that looks like a leak?

Slow GPU Despite a Good Card¶

If the GPU is available but performance is disappointing:

Check utilization — low utilization often means the GPU is waiting on the host or on data.
Check host-to-device transfers — frequent small copies can dominate runtime.
Check CPU preprocessing — data loading and augmentation can be the bottleneck.
Check batch size — tiny batches can underutilize the device.
Check precision and kernels — some operations fall back to slower paths.
Check for synchronization points — stray synchronizations can serialize work.

A GPU that says "10% utilization" is usually telling you the problem is not the GPU itself.

CUDA_VISIBLE_DEVICES¶

This environment variable controls which GPUs a process sees.

export CUDA_VISIBLE_DEVICES=0
export CUDA_VISIBLE_DEVICES=0,2

Use it to: - Restrict a process to specific GPUs - Mask GPUs for multi-GPU control - Debug visibility problems - Reproduce multi-GPU visibility issues

If a process cannot see a GPU you expect, this variable is a common culprit.

Container and WSL Issues¶

Docker¶

Docker containers often need explicit GPU access:

The NVIDIA driver must be available on the host.
The container runtime must have GPU access configured.
The container may need the CUDA libraries or a CUDA-enabled image.

If a containerized job cannot see CUDA, check: - Host driver availability - Container GPU runtime flags - Image CUDA compatibility - Whether the container sees nvidia-smi or the device nodes it needs

WSL¶

On WSL, GPU support depends on the Windows drivers and WSL configuration. If CUDA works on the Windows side but not in WSL, check: - Windows GPU driver status - WSL GPU support - Whether the WSL environment can reach the right device interfaces

Multi-GPU Visibility¶

For multi-GPU setups, confirm: - All GPUs appear in nvidia-smi - The framework sees the expected number of devices - No visibility variable is hiding devices - NCCL or the framework communication layer can initialize - Processes are not all trying to use the same device by default

A frequent mistake is assuming multi-GPU is configured just because the machine has multiple GPUs.

Library and Package Conflicts¶

CUDA-related installs can conflict when: - you install a framework CUDA build and also a separate CUDA toolkit - multiple packages expect different CUDA versions - a package was built against one CUDA/driver combination and run on another

Signs: - import succeeds but use fails - errors only in certain operations - confusing "version" messages

Practical mitigation: - keep one coherent CUDA stack per environment when possible - prefer the CUDA expected by the main framework you use - verify compatibility of any extra CUDA-dependent package before adding it

A Simple Diagnostic Script Pattern¶

A minimal CUDA diagnostic in Python:

import torch

print("torch:", torch.__version__)
print("cuda available:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("devices:", torch.cuda.device_count())
    print("current device:", torch.cuda.current_device())
    print("device name:", torch.cuda.get_device_name(0))
    print("memory allocated:", torch.cuda.memory_allocated(0))
    print("memory reserved:", torch.cuda.memory_reserved(0))

For non-PyTorch frameworks, use the equivalent device queries. The goal is the same: confirm visibility, count, and current memory footprint.

When to Suspect the Framework, Not CUDA¶

Sometimes the issue is not CUDA itself: - A framework was installed without CUDA support - You imported the wrong variant - You are on a build that targets CPU only - A model or op does not have a GPU kernel for your configuration - The framework is defaulting to CPU silently

If the system CUDA layer looks fine, check the framework's CUDA build and device placement explicitly.

Common Fixes, When Appropriate¶

Update or align the NVIDIA driver with the CUDA/toolkit needs
Use the CUDA version matched to the framework
Reduce batch size or memory pressure
Move tensors once, not repeatedly
Prefer contiguous tensors where the operation expects them
Avoid unnecessary host-device copies in hot loops
Use mixed precision or quantization if appropriate to the workload
Restrict visible devices to avoid accidental contention
In containers, ensure GPU access is properly configured

Only apply a fix after you have a hypothesis about which layer is failing.

gpu-basics — hardware concepts behind the symptoms
nvidia-cuda — CUDA stack and terminology
pytorch-gpu — PyTorch-specific device and memory checks
llm-hardware — GPU limits in LLM inference and fine-tuning
linux-commands — nvidia-smi, process inspection, and system-side checks
mlops — reproducible CUDA environments and environment drift

Official Documentation¶

NVIDIA driver and CUDA compatibility notes
nvidia-smi documentation for your driver version
Your framework's CUDA/device troubleshooting notes
Container GPU runtime docs for Docker, WSL, or your platform

When in doubt, narrow the layer first: system, driver, toolkit, framework, then code.

Last Updated¶

2026-09-06 — Initial version.