CyberFreezeDev Loading
  1. Home
  2. Blog
  3. R&D / Labs notes
  4. Ollama vs vLLM vs llama.cpp: which local LLM server to pick

R&D / Labs notes

Ollama vs vLLM vs llama.cpp: which local LLM server to pick

Ollama suits a laptop, llama.cpp suits one machine you want full control of, and vLLM suits a GPU server that many people use at once. Here is how they differ on batching, model formats, APIs and licences.

Sai Dharshan SFounder & AI consultant
Published
Reading time
5 min
Network cables plugged into a switch with green port lights
Network cables plugged into a switch with green port lights

Pick Ollama for a laptop or a single developer, llama.cpp when you want the same GGUF model files with full control on one machine (including one with too little GPU memory), and vLLM when many people or an application hit one GPU server at the same time. All three expose an OpenAI compatible API, so your application code barely changes; the real differences are how they batch requests, which model formats they load and which hardware they run on.

This is a Labs note for the technical lead who has to choose the inference server before the pilot starts. Versions move weekly, so treat the details below as correct on 27 October 2026.

Hands typing on a laptop at a bright white desk
On one laptop, Ollama is the shortest path from download to a working local model, and it listens only on the machine itself by default.

What each one is

llama.cpp is a C and C++ inference engine under the MIT licence. It runs models in the GGUF file format at 1.5 to 8 bit integer quantisation, on CUDA, Metal, Vulkan, HIP, SYCL and plain CPU builds. Its llama-server program serves an HTTP API. It can split a model between GPU and CPU when the model is larger than your VRAM.

Ollama wraps model download, storage and serving into one tool, also MIT licensed, for macOS, Windows, Linux and Docker. Its README lists llama.cpp as a supported backend. It imports a GGUF file or a Safetensors directory through a short Modelfile.

vLLM is a serving engine under Apache 2.0, built for throughput on shared servers. It runs on Linux (Windows only through WSL), needs an NVIDIA GPU of compute capability 7.5 or higher for CUDA, and also supports AMD and Intel GPUs. It can split one model across several GPUs with tensor and pipeline parallelism.

Ollama vs vLLM vs llama.cpp side by side

Ollama llama.cpp (llama-server) vLLM
Licence MIT MIT Apache 2.0
Model formats GGUF; Safetensors via a Modelfile GGUF (convert from Hugging Face with its script) Hugging Face weights, FP8, INT8, INT4, GPTQ, AWQ; GGUF via a plugin, marked highly experimental
OpenAI API A subset at localhost:11434/v1: chat, completions, embeddings, models, responses (non-stateful) Chat completions, responses and embeddings routes Completions, chat, responses, embeddings, audio transcriptions; also the Anthropic Messages API
Requests at once One per model by default; raise OLLAMA_NUM_PARALLEL Several slots, continuous batching on by default Continuous batching with PagedAttention
Runs on macOS, Windows, Linux Almost anything, GPU or CPU Linux servers with supported GPUs
GPU too small Loads part on GPU, part in system memory Splits layers between GPU and CPU Built for models that fit in GPU memory

How do they handle many users at once?

A GPU generating text for one person spends most of each step reading weights from memory. Batching puts tokens from several people into the same step, so each read of the weights serves more users. This is where the three differ most.

  • Ollama processes one request per model at a time by default and queues up to 512 more. You can raise OLLAMA_NUM_PARALLEL, but its FAQ warns that memory scales with the number of parallel requests times the context length.
  • llama.cpp gives llama-server several slots, with continuous batching enabled by default and one KV cache buffer shared across them. Good for a small team on one box.
  • vLLM was designed around this problem. PagedAttention stores the KV cache in pages so little memory is wasted, continuous batching adds new requests between steps, and prefix caching reuses work when every request starts with the same long system prompt.

We have not published a benchmark of the three, so we give no throughput figures here. Test with your own prompts, context lengths and number of users before you size hardware.

A man working on a laptop at a wooden table by a window
When a whole team sends prompts at the same time, batching on the server matters more than how easy it was to install.

Which serving stack for your hardware?

  1. A laptop or one developer's workstation: Ollama. Install, pull a model, call localhost. It binds to 127.0.0.1 by default, so nothing is exposed to the network until you change OLLAMA_HOST. Our lab machine is an 8 GB laptop GPU, which is the kind of card this suits.
  2. One GPU office server for a small team: llama.cpp or vLLM. If the model only fits by spilling onto the CPU, or the box is Windows or a Mac, use llama-server. If it is a Linux server with an NVIDIA card and the model fits in VRAM, vLLM will batch better as use grows.
  3. Many concurrent users, or an app calling the model: vLLM. Continuous batching, prefix caching and multi-GPU parallelism are what this load needs. Plan for Hugging Face format weights or AWQ, GPTQ and FP8 checkpoints rather than GGUF.

Because all three speak the OpenAI format, you can start a pilot on Ollama and move it to vLLM later by changing a base URL and a model name. Test tool calling and structured output again after the move, as each server implements them differently.

What do the licences mean for a business?

MIT and Apache 2.0 both allow commercial use, modification and private deployment. Apache 2.0 adds an express patent licence from contributors. None of the three charges per user or per token.

The model is a separate question. Each open-weight model has its own licence, from Apache 2.0 to custom terms with user limits and acceptable use policies. Read the model card before you put it in front of staff; our guide to running an on premise LLM covers model licences and hardware tiers.

Before you open the port to the office

A local model server is a database of your company's prompts. vLLM's own docs say its --api-key flag only authenticates requests and should not be the only protection. For any of the three, put the server behind a reverse proxy with TLS and staff sign-in, keep it on the internal network, and log who asked what. That is the difference between a demo and on premise AI your compliance lead will sign off.

Choosing for your own team

If you are deciding between these for a real workload, our engineers can size the model, pick the server and set it up on your hardware or a private cloud through our private AI deployment service. Bring your number of users, document lengths and the GPU you have. Book a consultation.