CyberFreezeDev
Loading
Storing model weights in fewer bits to cut memory, at a small cost in accuracy.
Weights shipped at 16 bits are converted to 8, 4 or fewer. A 7 billion parameter model drops from about 14 GB to about 4 GB.
A model is mostly a very large table of numbers (weights). They are usually trained and shipped as 16-bit floating point values, two bytes each. Quantization converts them to 8-bit, 5-bit, 4-bit or lower representations.
The arithmetic is simple: a 7 billion parameter model at 16 bits needs about 14 GB just for weights. At 4 bits it needs roughly a quarter of that, around 4 GB, plus some overhead. That is the difference between needing a data-centre card and fitting on an 8 GB laptop GPU with room for the context window.
Weights are split into small blocks, such as 32 or 128 values.
Weights are split into small blocks, for example 32 or 128 values each.
Each block gets a scale factor that maps its values onto a few integer levels.
Each block gets a scale factor so its values can be mapped onto a small set of integer levels.
Values are rounded and stored with the scale; GPTQ and AWQ protect the sensitive weights.
Values are rounded to those levels and stored with the scale. Methods like GPTQ and AWQ use sample data to decide which weights need more care.
During inference the runtime expands blocks back into usable numbers on the GPU.
At inference time the runtime expands blocks back to usable numbers inside the GPU kernel.
A firm can run an assistant on one workstation instead of renting cloud GPUs.
Quantized open-weight models let a firm run an assistant on one workstation instead of renting cloud GPUs.
Models on Jetson boards and other edge hardware almost always run quantized.
Vision and language models on Jetson boards or other edge AI hardware almost always run quantized.
Smaller weights move through memory faster, so tokens per second usually rise.
Smaller weights move through memory faster, so tokens per second usually go up on the same card.
An 8 billion parameter model does not fit our 8 GB laptop GPU at 16 bits. At 4 bits it is about 5 GB and loads fully.
Our own lab machine has an RTX 3070 Ti laptop GPU with 8 GB of VRAM. An 8 billion parameter model at 16 bits does not fit. The same model as a 4-bit GGUF file for llama.cpp or Ollama is roughly 5 GB, which loads fully on the card and leaves space for a working context. Any local model result we publish in Labs states the hardware and quantization level it was run with.
Quality drops gently to about 4 bits, then faster. 2 and 3-bit small models often break.
Quality drops gently down to around 4 bits and then falls faster. 2 and 3-bit versions of small models often break on reasoning and structured output.
Test the quantized model on your real task with a fixed evaluation set.
Test the quantized model on your actual task with a fixed evaluation set, especially extraction and maths.
GGUF suits llama.cpp; GPTQ and AWQ usually suit GPU runtimes like vLLM. Match your runtime.
GGUF is for llama.cpp and tools built on it; GPTQ and AWQ files are typically served with GPU runtimes such as vLLM. Pick the format your runtime supports.
It does not remove knowledge on purpose, but rounding adds small errors everywhere. The effect is usually a slight drop in accuracy, larger on hard reasoning tasks and on smaller models.
For local LLMs a 4-bit or 5-bit variant (such as GGUF Q4_K_M or Q5_K_M) is a common starting point. Move up to 8-bit if your tests show quality problems and memory allows.
Last updated