Can you run an LLM on a CPU? Sizes, RAM and speed

Yes, you can run an LLM on a CPU. llama.cpp and Ollama run quantised models with no GPU at all, and a 4-bit model of up to about 8 billion parameters is practical on a recent laptop or desktop with 16 GB of RAM. Speed depends on memory bandwidth rather than core count: for every word it writes, the model reads its weights from RAM, so a model twice the size runs at roughly half the speed.

That rule tells you which sizes are worth trying, how much RAM to buy and when a GPU pays off.

How does an LLM run without a GPU?

llama.cpp is written in plain C and C++ with no dependencies, and uses the vector instructions of modern processors: AVX, AVX2, AVX512 and AMX on Intel and AMD, NEON on Arm. It stores weights at 1.5 to 8 bits, which is what makes inference on a CPU possible at all. Ollama runs the same GGUF model files. On a machine without a supported GPU it loads the model into system memory, and ollama ps shows "100% CPU".

A note on Macs: Ollama uses the GPU on Apple M series Macs, so they are not CPU-only machines. Intel Macs run on the CPU.

Why does memory bandwidth decide the speed?

An LLM works in two phases. Prefill reads your prompt in one batch, and each weight fetched from memory is used for every prompt token, so it is limited by arithmetic and gets faster with more cores. Decode writes the answer one token at a time, and every token needs the weights read from memory again. Arm's documentation puts it as a formula:

decode tokens per second = effective memory bandwidth ÷ bytes read per token

Arm measured this on a Raspberry Pi 5 with llama.cpp, a machine that sustains 13.8 GB/s:

Each result is 84 to 86 percent of the bandwidth ceiling. Two more findings matter for any CPU. Decode was fastest at two threads; four threads made it 19 percent slower, because the extra threads only wait on the same memory. Prefill, by contrast, was 3.1 times faster at four threads than at one.

What speed can a desktop or server expect?

The same formula gives a ceiling for other machines. These are our sums from published specifications, not benchmarks we ran.

A current desktop processor such as Intel's Core Ultra 7 265K has two memory channels of DDR5 at up to 6,400 MT/s. Supermicro's tables put one such channel at 51.2 GB/s, so two give a theoretical 102.4 GB/s. An AMD EPYC 9005 server processor has twelve channels, 614.4 GB/s per socket.

Real speeds sit below these: on the Raspberry Pi, llama.cpp reached about 85 percent of the bandwidth the machine actually sustains, which is itself below the theoretical figure. Decode also slows as the conversation grows, because the model reads its saved context for each token too.

Mixture-of-experts models change the sums. Qwen3 30B A3B has 30.5 billion parameters but routes each token through only 8 of its 128 experts, so 3.3 billion parameters are active per token. RAM must hold the whole 19 GB file, but each token reads only the active part, so it should write far faster than a 32B dense model on the same CPU. Measure it before you rely on it.

How much RAM do you need?

Add the model file, room for the context and what your operating system and other programs use. Sizes are Ollama's 4-bit downloads of Qwen3; the RAM column is our rule of thumb.

Ollama's FAQ notes that memory grows with context length times the number of requests served in parallel. Setting OLLAMA_KV_CACHE_TYPE=q8_0 stores that context memory at 8 bits, about half the default, when Flash Attention is on.

Getting the most out of a CPU

1. Use 4-bit files. Q4_K_M is the usual start; our guide to GGUF, AWQ and GPTQ explains the names.
2. Fill every memory channel. Bandwidth is per channel, so a dual-channel desktop with one memory module uses one channel: half the theoretical bandwidth.
3. Test thread counts. For chat, more threads can make decode slower; for summarising long documents, all cores help prefill. In llama.cpp, set -t explicitly.
4. Keep the context short. Send the pages the model needs, not the whole file.
5. Try a mixture-of-experts model when you need more quality than 8B gives.

When should you buy a GPU instead?

A CPU is fine for one person, small models and overnight batch jobs. Buy a GPU when several people will use the model at once, when prompts are long documents, or when you need a 14B or larger model at a speed people will wait for. The gap is bandwidth: an NVIDIA L40S has 864 GB/s and an H200 4.8 TB/s, against about 100 GB/s for a dual-channel desktop. To size the card, start with our guide to how much VRAM an LLM needs.

Planning hardware for local models

Our lab tests open-weight models on modest hardware we own, and our local LLM research records what we find. If you are deciding between a CPU server, a GPU workstation or a rented machine, our GPU servers and AI hardware service starts from your models and users. Book a consultation.