CyberFreezeDevVRAM
The GPU's own memory. The model must fit, and it is often the first limit.
Read more about VRAM
The GPU's own memory. The model weights and working data must fit in it. See VRAM; it is often the first limit you hit.
- Home
- AI glossary
- GPU
What is a GPU, and why does AI need one?
A graphics chip that does thousands of small calculations at once, which neural networks need.
Definition
A CPU has a few strong cores; a GPU has thousands of simple ones. Neural networks are mostly matrix maths, which suits GPUs well.
Read more about Definition
A GPU (graphics processing unit) is a chip designed for massively parallel work. A CPU has a handful of powerful cores tuned for varied tasks one after another. A GPU has thousands of simpler cores that apply the same operation to large blocks of numbers at once.
A neural network is mostly matrix multiplication: inputs times weights, layer after layer. That maps directly onto a GPU, which is why training and inference for anything beyond small models runs on GPUs or similar accelerators.
What matters on a GPU for AI
Memory bandwidth
How fast the GPU reads its memory, often the real limit on LLM text speed.
Read more about Memory bandwidth
How fast the GPU can read its memory. For LLM text generation this often limits speed more than raw compute.
Tensor cores and precision
Special cores for fast low-precision maths, which quantized models use.
Read more about Tensor cores and precision
Recent NVIDIA GPUs have tensor cores for fast low-precision maths (FP16, BF16, INT8, FP8 on newer parts), which quantized models use.
Software support
Most AI frameworks target NVIDIA CUDA first; AMD ROCm and Apple Metal are catching up.
Read more about Software support
NVIDIA's CUDA platform is what most AI frameworks target first. AMD's ROCm and Apple's Metal work for a growing set of tools.
Where it matters in business
Private LLMs
Running an open-weight model on your own server needs a GPU sized to it.
Read more about Private LLMs
Running an open-weight model on your own server needs at least one GPU sized to the model.
Image and video generation
Diffusion models are compute heavy; a GPU turns minutes into seconds per image.
Read more about Image and video generation
Diffusion models are compute heavy; a GPU turns minutes into seconds per image.
Vision on site
Camera analytics on edge boards like Jetson use a small built-in GPU.
Read more about Vision on site
Camera analytics on edge boards such as NVIDIA Jetson use a small integrated GPU.
A practical example
Our 8 GB laptop GPU runs quantized 7 to 8 billion parameter models and image work, but not 70 billion ones, and it throttles under long jobs.
Read more about A practical example
Our lab machine includes an RTX 3070 Ti laptop GPU with 8 GB of VRAM. That is enough to run quantized language models in the 7 to 8 billion parameter range and to upscale or generate images, but not to load a 70 billion parameter model. Under long batch jobs a laptop GPU also heats up and the driver lowers its clock speed to protect itself, so sustained throughput is lower than the spec sheet suggests. Server cards are built for continuous load.
Common mistakes
Buying on compute numbers alone
A fast card with too little VRAM cannot load the model. Check memory first.
Read more about Buying on compute numbers alone
A fast card with too little VRAM cannot load the model at all. Check memory first.
Ignoring cooling and power
Multi-GPU servers need airflow and electrical capacity, or throttling quietly cuts speed.
Read more about Ignoring cooling and power
Multi-GPU servers need proper airflow and electrical capacity. Throttling quietly cuts performance.
Buying before measuring
Rent a cloud GPU for a week with your real workload, then decide.
Read more about Buying before measuring
Rent a cloud GPU for a week, run your real workload, then decide whether to buy.
Related terms and pages
Questions
Can AI run without a GPU?
Small models can run on a CPU, and tools such as llama.cpp make that practical for light use. It is much slower, so for anything interactive or high-volume a GPU or another accelerator is the normal choice.
Is a gaming GPU good enough for business AI?
For development, testing and small internal tools, often yes. For continuous production load, data centre cards offer more VRAM, better cooling, error-correcting memory and licensing suited to servers.
Do I need NVIDIA specifically?
Not always, but most AI software supports NVIDIA's CUDA first, so it is the lowest-friction option. Check that every tool in your stack supports the alternative before choosing AMD or another vendor.
Last updated