CyberFreezeDevFits
Under 90 percent of the card, with room for longer prompts.
Read more about Fits
The estimate is under 90 percent of the card, leaving some room for longer prompts and driver overhead.
- Home
- Free tools
- VRAM calculator for LLMs
VRAM calculator for running an LLM locally
Check whether an LLM fits your GPU: weights, KV cache and runtime added up in your browser.
Estimate GPU memory
Starts with Llama 3.1 8B at Q4_K_M and 8,192 tokens of context.
What uses the memory
Memory goes to the weights, set by parameters and bit width; the KV cache, which grows with context; and about 1 GB for the runtime.
Read more about What uses the memory
Three things share the card when a model runs.
The weights. Every parameter is stored at some number of bits. At 16-bit, each billion parameters takes about 2 GB; at a 4-bit format such as Q4_K_M it is a little over 0.6 GB. This is the part that quantization shrinks.
The KV cache. For every token in the context, each layer stores a key and a value vector so the model does not recompute them. The size per token is fixed by the model's shape: 2 × layers × key/value heads × head size × bytes. For Llama 3.1 8B that is 128 KB per token at 16-bit, so 8,192 tokens add about 1.1 GB. Long documents and several users at once multiply it.
The runtime. CUDA context, compute buffers and the serving software itself. The calculator allows about 1 GB, which is a rough figure; it varies by runtime and settings.
For what these terms mean, see VRAM and GPU in our glossary.
Reading the result
Tight
It may load, but a long prompt can overflow. Shorten context or use a smaller format.
Read more about Tight
It may load, but a long prompt can run out of memory. Shorten the context, use an 8-bit KV cache or pick a smaller format.
Does not fit
Offload layers to the CPU at a large speed cost, or use a bigger card.
Read more about Does not fit
Runtimes such as llama.cpp can keep some layers on the CPU, which works but is much slower. For a team, a bigger card or a second GPU is the usual answer.
Where the figures come from
Model shapes come from each config.json on Hugging Face and bits per weight from llama.cpp. The list starts at 8 GB because we run that card.
Read more about Where the figures come from
Layer counts, key/value heads and head sizes come from each model's published config.json on Hugging Face. Bits per weight for the GGUF formats are the averages llama.cpp reports for its quantization types; real files vary a little because some tensors are kept at higher precision. Card sizes are as sold, in GiB.
We run models on an 8 GB RTX 3070 Ti laptop GPU in CyberFreeze Labs, which is why 8 GB is the first card on the list. If you are sizing a server for a team, our GPU servers and AI hardware and private AI deployment services start from your workload, not a price list.
Questions
How much VRAM do I need to run an 8B model?
At a 4-bit format such as Q4_K_M, about 5 GB for the weights plus about 1 GB of KV cache at 8,000 tokens and a runtime allowance, so an 8 GB card is enough for one user. At 16-bit the weights alone are about 16 GB.
Why does my model run out of memory with long documents?
The KV cache grows with every token of context. A model that loads fine with a short prompt can fail when you paste a long contract. Lower the context length, use an 8-bit KV cache, or move to a card with more memory.
Does the calculator work for Mac or CPU inference?
The weight and KV cache figures apply anywhere, but the card list assumes an NVIDIA or similar GPU with its own memory. On a Mac with unified memory, compare the total against the memory the system lets the GPU use.
Last updated