CyberFreezeDev Loading
  1. Home
  2. Blog
  3. How-to guides
  4. How much VRAM do you need to run an LLM?

How-to guides

How much VRAM do you need to run an LLM?

GPU memory for an LLM is the weights plus the KV cache plus a runtime allowance, and all three can be worked out from the model's config file. Here is the arithmetic for 8B, 32B and 70B models and what fits on 8, 16, 24, 48 and 80 GB cards.

Sai Dharshan SFounder & AI consultant
Published
Reading time
5 min
A colourful close up of the circuitry on a silicon chip
A colourful close up of the circuitry on a silicon chip

You need enough VRAM for three things: the model weights (parameters × bytes per weight), the KV cache that holds the conversation (it grows with every token and every user), and roughly 1 GB for the runtime. An 8B model at 4-bit fits on an 8 GB card, a 32B model at 4-bit fits on 24 GB with a modest context, and a 70B model needs 48 GB at 4-bit or 80 GB to leave room for several users.

Below is the arithmetic, so you can check any model yourself. Our VRAM calculator does the same sums in your browser. It shows decimal GB (1,000,000,000 bytes) and uses 4.85 bits for Q4_K_M, so for the 8B example below it reads about 6.9 GB where this post says 6.6 GiB: the same amount of memory in a different unit.

Blue memory modules in the slots of a computer motherboard
The weights take most of the memory, and their size is set by two numbers: the parameter count and the bits stored per weight.

How do you work out the memory for the weights?

NVIDIA's inference guide puts it simply: a 7B model at 16-bit takes about 7 billion × 2 bytes, or 14 GB. The general form is:

weights in bytes = parameters × bits per weight ÷ 8

Quantization changes the bits per weight. llama.cpp publishes the average for each of its GGUF formats:

Format Bits per weight Bytes per billion parameters
F16 / BF16 16 about 2.0 GB
Q8_0 8.50 about 1.06 GB
Q4_K_M 4.89 about 0.61 GB
IQ4_XS 4.46 about 0.56 GB

The figures are slightly above the nominal 8 or 4 bits because each block of weights also stores a scale. Below we work in GiB, the unit llama.cpp reports.

How big is the KV cache?

For every token in the context, every layer stores a key and a value vector. NVIDIA's formula:

KV cache per token = 2 × layers × key/value heads × head size × bytes

Modern models use grouped query attention, so use the key/value head count from the model's config.json, not the query head count. Then multiply by the context length and by the number of conversations running at once.

At 16-bit (2 bytes), using each model's published config:

Model Layers KV heads Head size Per token 8,192 tokens 32,768 tokens
Llama 3.1 8B 32 8 128 128 KiB 1.0 GiB 4.0 GiB
Qwen3 32B 64 8 128 256 KiB 2.0 GiB 8.0 GiB
Llama 3.3 70B 80 8 128 320 KiB 2.5 GiB 10.0 GiB

Working for the 8B row: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token; 8,192 tokens × 128 KiB = 1 GiB.

This is why a model that loads fine can run out of memory when someone pastes a long contract, and why ten users at 8,192 tokens on the 70B model need 25 GiB of cache on top of the weights. An 8-bit KV cache (--kv-cache-dtype fp8 in vLLM) roughly halves it.

Worked examples: 8B, 32B and 70B

One user, 8,192 tokens of context, 1 GiB runtime allowance.

8B (Llama 3.1 8B). llama.cpp lists 4.58 GiB at Q4_K_M, so 4.58 + 1.0 + 1.0 = 6.6 GiB. At Q8_0: 7.95 + 2.0 = 10.0 GiB. At 16-bit: 14.96 + 2.0 = 17.0 GiB.

32B (Qwen3 32B, 32.8B parameters). Q4_K_M: 32.8 × 4.89 ÷ 8 = 20.1 GB, which is 18.7 GiB, plus 3.0 = 21.7 GiB. Q8_0: 32.5 + 3.0 = 35.5 GiB. 16-bit: 61.1 + 3.0 = 64.1 GiB.

70B (Llama 3.3 70B, 71B parameters in its file metadata). Q4_K_M: 71 × 4.89 ÷ 8 = 43.4 GB, which is 40.5 GiB, plus 3.5 = 44.0 GiB. IQ4_XS: 36.9 + 3.5 = 40.4 GiB. Q8_0: 70.3 + 3.5 = 73.8 GiB. 16-bit: 132.2 + 3.5 = 135.7 GiB.

Blue network cables plugged into a server switch
Once a 70B model leaves room for only one or two long chats, a bigger card or a second GPU is usually the next step.

What fits on 8, 16, 24, 48 and 80 GB cards?

We call it a fit when the total is under 90 percent of the card, which leaves room for longer prompts and driver overhead. "Tight" means it may load but a long prompt can fail.

Card memory (example card) 8B 32B 70B
8 GB (our lab's RTX 3070 Ti laptop GPU) Q4_K_M at 8k context No No
16 GB (RTX PRO 2000 Blackwell) Q8_0 No: 4-bit weights alone are 18.7 GiB No
24 GB (RTX PRO 4000 Blackwell) 16-bit Q4_K_M at about 4k context; 8k is tight No
48 GB (L40S, RTX PRO 5000 Blackwell) 16-bit, several users Q8_0 IQ4_XS with a short context; 8k is tight
80 GB (H100 SXM) 16-bit, many users 16-bit Q4_K_M with room for several users; Q8_0 is tight

A 70B model at 16-bit needs two 80 GB cards or more. Runtimes such as llama.cpp can also keep some layers in system RAM when the GPU is too small; it works, but it is much slower, so treat it as a test setup rather than a service for a team. For the terms, see VRAM in our glossary.

A quick checklist before you buy a card

  1. Pick the model and read its config.json: parameters, layers, KV heads, head size.
  2. Pick the format. 4-bit is the usual starting point on one card; check quality on your own documents.
  3. Decide the longest context and the number of people using it at once. Multiply.
  4. Add 1 GiB, keep 10 percent spare, and compare with the card.

These are estimates from the published formula, not measurements of ours. Real usage varies with the runtime, its settings and the model's attention design, so test on the hardware before you commit to it.

Sizing a server for your team

If you are choosing hardware for a team rather than one person, the KV cache for concurrent users usually decides the card, not the weights. Our GPU servers and AI hardware service starts from your model, document lengths and number of users, and works back to the build. Book a consultation.