CyberFreezeDev
Loading
We test which business tasks an open-weight model handles on an 8 GB laptop GPU.
Starting on one 8 GB laptop GPU: if a task works there it will work on a bigger server, and if not, we record why.
Which everyday business tasks can a local LLM handle well enough on modest hardware you own, and where does it stop being worth it?
By local LLM we mean an open-weight model (families such as Llama, Qwen, Mistral and Gemma) running on a machine you control, with no prompt or document sent to an outside API. We deliberately start small: one laptop GPU with 8 GB of video memory. If a task works there, it will work on the workstation or server a client is more likely to buy. If it fails there, we want to know exactly why: memory, speed, or the model simply not being good enough.
The answer feeds our private AI deployment work and the hardware advice we give in GPU servers and AI hardware builds. To check whether a model fits your own card, use our VRAM calculator.
Three reasons clients ask us about local models, in the order we hear them.
A model on your own server removes one data processor and one cross-border transfer.
Clinics hold patient notes, CA firms hold client books and bank statements, HR teams hold ID documents. India's Digital Personal Data Protection Act 2023 and UK GDPR both make you answerable for every processor that touches personal data. A model on your own server removes one processor and one cross-border transfer from that list.
Local models cost hardware and upkeep, not tokens. Which is cheaper depends on volume.
Hosted APIs bill per token. A local model costs the hardware, electricity and upkeep, whether it reads ten documents a day or ten thousand. Which is cheaper depends on volume, and we want real measurements to make that call for a client instead of guessing.
Once installed, a local model has no outside dependency, even with patchy internet.
A factory floor, a clinic with patchy internet or a field office still needs the tool to answer. A local model has no outside dependency once it is installed.
A model file on your disk behaves the same next year as today.
Hosted models change and get retired on the provider's schedule. A model file on your disk behaves the same next year as it does today, which matters when a workflow has been checked and signed off.
One machine, chosen because it is the low end of what a small office might buy. Exact software versions are recorded with every result we publish.
Speed numbers alone are misleading. A fast model that returns broken JSON is useless in a workflow, so every run is scored on speed, memory and task quality together.
| Metric | How we measure it | What counts as success |
|---|---|---|
| Time to first token | Seconds from sending the prompt to the first output token, at short and long prompt lengths | Short enough that a person waiting at a screen does not give up; judged per task |
| Generation speed | Output tokens per second, averaged over the whole task set, not the best run | Fast enough to clear the day's volume for that task inside working hours |
| Peak video memory | Highest VRAM use reported by nvidia-smi during a run, including the context cache | Fits fully on the GPU with no layers spilled to system RAM |
| Format validity | Share of outputs that parse as JSON and pass the task's schema | Every output in the set parses; failures are retried once and logged |
| Task accuracy | Field by field comparison against answers a person wrote for each test item | Matches what a trained person would accept without edits |
| Grounding | For question answering over documents: does every claim cite a passage that actually contains it | No answer without a source passage; unsupported answers are counted as failures |
| Sustained run | The same task set looped for a long session, watching speed and GPU temperature | Speed holds steady; no crash or slowdown from heat |
Task sets: GST invoice field extraction, short answers from a fixed document set, and sorting customer messages into categories. Test items are written by us or taken from public samples.
The first two points are public arithmetic and documented behaviour. The last two are our working hypotheses, not results.
At 16-bit an 8B model needs about 16 GB; at 4-bit, roughly 4 to 5 GB.
Weights take roughly parameters multiplied by bytes per parameter. An 8 billion parameter model at 16-bit precision needs about 16 GB just for weights, so it cannot fit on an 8 GB card. At 4-bit quantization the same model is closer to 4 to 5 GB, which leaves room for the context.
The KV cache grows with every token, so long documents can run out of memory.
The key and value cache grows with every token of context. A model that fits comfortably with a one page prompt can run out of memory on a forty page contract. llama.cpp can offload some layers to system RAM, but speed then drops sharply.
Hypothesis: a quantized 7 to 8B model with a strict schema and retrieval handles extraction and summaries.
We expect a quantized 7 to 8 billion parameter model, given a strict output schema and the right passages from retrieval-augmented generation, to handle extraction, classification and summaries of known document types. This is what we are testing first.
Hypothesis: multi-step judgement likely needs larger models than an 8 GB card can hold.
Multi-step judgement, such as reviewing an unusual contract clause, likely needs larger models than an 8 GB card can hold. Finding that boundary task by task is the point of this track.
Every task set at 4-bit and 8-bit, logging speed, memory, accuracy and versions.
Run every task set across the model families above at 4-bit and 8-bit, and record speed, memory and accuracy with versions and dates.
Several staff on one local model at once, including vLLM on larger cards.
Test what happens when several staff use one local model at once, the realistic office case, including serving engines such as vLLM on larger cards.
The strongest setups move into the document AI and multi-agent tracks for real work.
Feed the strongest local setups into the document AI track and the multi-agent track, where they do actual work instead of answering test prompts.
Lab notes will be published in the R&D / Labs notes section of our blog. Nothing below is a result until it appears there with its run details.
| Note | Published when | What it contains |
|---|---|---|
| Method note | Before any scores | The question, the rig above, the three task sets, how each metric is computed and what counts as a pass |
| Baseline, per model family | When a family has finished its 4-bit and 8-bit runs on every task set | Time to first token, tokens per second, peak VRAM, format validity and accuracy per task, with runtime version, model file, quantization, driver version and run date |
| Prompts and per-item scores | With each baseline note | The prompts and test items, and the score for every item, so a run can be repeated on any 8 GB card |
| Multi-user serving | After the single-user baselines | Throughput and response time as concurrent users are added, with the serving engine and GPU named |
For general open-ended questions, usually not on small hardware. For a defined task, such as pulling fields from invoices or sorting support messages, a local model can be good enough. This track exists to say which tasks, with measurements.
It depends on model size, precision and how much text you send at once. As a rule of thumb from public figures, an 8 billion parameter model at 4-bit fits in 8 GB with modest context; larger models or many simultaneous users need more memory. We size this per project in GPU servers and AI hardware builds.
No single choice makes you compliant. Running locally removes an outside processor and a data transfer, which simplifies the picture, but you still need consent, purpose limits, access control and retention rules.
Not yet. We only publish numbers from runs we actually did, with the hardware, software versions and date. They will appear as lab notes in the R&D / Labs notes section of our blog.
Last updated