Multi-agent systems on your own servers: a practical guide

You can run a multi-agent system entirely on your own servers: an open-weight model served by vLLM, llama.cpp or Ollama, an orchestrator such as LangGraph, CrewAI, Microsoft Agent Framework or n8n pointed at that local endpoint, and MCP servers that give each agent narrow access to your internal systems. Installing it is the easy part; what decides success is keeping a small local model on track over several steps, fitting the agents on one GPU, and making sure no framework sends traces or telemetry out of the building.

This guide is for owners and engineering leads with a job like this: one agent reads invoices, one checks them against purchase orders, one drafts the supplier reply, and no document may leave the office. For the definition, see our multi-agent system glossary entry; this post is about running one.

The architecture, layer by layer

1. Model server. One open-weight model behind an OpenAI-compatible API. Agents share it; roles differ by prompt and tools.
2. Orchestrator. Decides which agent runs next, passes results, enforces step limits and stores state so a failed run can resume.
3. Tools. Each agent gets a few typed actions, such as read_invoice(file) or draft_supplier_email(invoice_id, issue), usually exposed through MCP servers.
4. Deterministic code. Arithmetic, matching and lookups with one right answer run as functions or SQL, not model calls.
5. Approval and logs. A person approves anything that changes a record or reaches a supplier, and every tool call goes to a log you own.

In the invoice example, the reader agent fills a fixed JSON schema, code matches it to the purchase order, the drafting agent writes only for flagged invoices, and accounts approves before sending. Two model roles and one code step, not five agents.

Frameworks that run fully local, and what to switch off

All the common frameworks can call a model on your own network. Several also send data outward by default, which matters more than any feature list here.

The AutoGen repository is archived and points new users to Microsoft Agent Framework, so do not start a new build on AutoGen.

Two more leaks: the OpenAI Agents SDK's hosted MCP tool has OpenAI's servers call a publicly reachable MCP server, so use its stdio or Streamable HTTP clients instead; and n8n's human review step can route approvals through Slack, Gmail or Teams, while only its built-in Chat channel stays inside n8n.

Tool calling on vLLM, llama.cpp and Ollama

• vLLM guarantees a parsable call when you name the tool or set tool_choice to required, but its docs add: "not a high-quality one". Automatic tool choice needs --enable-auto-tool-choice and the --tool-call-parser for your model family. Without strict mode, arguments "may occasionally be malformed".
• llama.cpp has native handlers for families such as Llama 3.x, Qwen 2.5, Mistral Nemo and gpt-oss, and a less efficient generic fallback. Its docs warn that KV cache quantization as aggressive as q4_0 can substantially degrade tool calling.
• Ollama supports tool calls, parallel calls and a format field that takes a JSON schema. Its docs suggest also putting the schema in the prompt and setting temperature to 0.

Whichever you use, validate every argument in your tool code: valid JSON can still name the wrong invoice.

Which open-weight models to shortlist

The Berkeley Function Calling Leaderboard (V4, last updated 12 April 2026) scores single tool calls and multi-turn tasks separately, and the gap is what matters for local builds.

Small models pick the right tool for one request almost as well as the largest ones; over several turns they fall away. Another candidate is OpenAI's gpt-oss-20b: Apache 2.0, 21B parameters with 3.6B active, runs within 16 GB, with native function calling and structured outputs. Treat leaderboard scores as a shortlist; your own test cases decide.

Why small local models lose the plan

Anthropic's engineers put it plainly: "Agents are stateful and errors compound." A study of more than 1,600 annotated traces from seven multi-agent frameworks (Cemri et al., "Why Do Multi-Agent LLM Systems Fail?") sorted the failures into 14 modes in three groups: system design, misalignment between agents, and task verification. At nine in ten per step, a ten step plan comes out fully right about a third of the time. What works:

1. Use fewer agents. Anthropic measured its multi-agent research system at about 15 times the tokens of a chat. On your own GPU, tokens are queue time and memory.
2. Give each role one job and few tools. OWASP's guidance on excessive agency is to limit tools to "the minimum necessary".
3. Move fixed steps into code. Microsoft's framework docs say: "If you can write a function to handle the task, do that instead of using an AI agent." Matching an invoice total to a purchase order is a query.
4. Let code hold the plan. Call the model one step at a time, constrain output to a schema, and cap steps per task.
5. Put a person before anything final. OWASP recommends human approval for high-impact actions; LangGraph and n8n both have a built-in pause for it.
6. Keep a test set. Real past cases, rerun whenever the model, a prompt or a tool changes.

Several agents on one GPU

Serve one model with different prompts per agent, then budget memory for context: each agent's conversation sits in the KV cache.

A worked example for Qwen3-8B (arithmetic from its config and llama.cpp's quantization table, not a benchmark): at Q4_K_M, about 4.89 bits per weight, the 8.2B weights take about 5 GB. Each token of context needs 2 × 36 layers × 8 KV heads × 128 dimensions × 2 bytes, about 144 KB at 16-bit, so an agent holding 8,000 tokens adds about 1.2 GB. Three agents at 8,000 tokens each need about 3.5 GB on top of the weights, over 8 GB before runtime overhead. One at a time fits.

• Ollama handles one request per model at a time by default (OLLAMA_NUM_PARALLEL is 1), so agents queue, and raising it multiplies context memory. The default context is 4,096 tokens, which agent prompts with tool results can exceed; set OLLAMA_CONTEXT_LENGTH.
• vLLM reserves 92% of GPU memory per instance by default. Two instances on one GPU need --gpu-memory-utilization lowered, for example to 0.5 each. Prefix caching reuses work when requests share a long prefix, and one base model can serve several LoRA adapters if you fine-tune per role.
• llama.cpp batches continuously by default and sizes parallel slots automatically; set --parallel and --ctx-size so you know what each agent gets.

Connecting agents to your systems with MCP

The Model Context Protocol (current revision 2026-07-28, under the Agentic AI Foundation at the Linux Foundation since December 2025) gives every framework above a common way to call tools, over stdio for a server next to the agent or Streamable HTTP for one on your network. For a build that keeps data in:

• Keep servers inside your network. CrewAI's docs warn that SSE servers are open to DNS rebinding and advise binding local servers to 127.0.0.1 and validating Origin headers.
• Put permissions in the server, not the prompt. The MCP spec says hosts must get explicit user consent before invoking any tool, and that tool descriptions are untrusted unless they come from a trusted server.
• One server per system. Accounts, CRM and the document store each get their own, with read-only tools by default and draft-style writes.

This is how we build MCP servers for our own products: the tools on Fint's server can only propose records, and nothing is recorded until a person confirms it.

What CyberFreezeDev does here

We install open-weight models on hardware you control, served by vLLM, llama.cpp or Ollama behind an OpenAI-compatible API, as part of private AI deployment. On top of that we build the agents in AI agent development: narrow tools, per tool permissions, a draft and approve step, shadow mode on live work, and a test set built from 30 to 100 of your past cases. In our own engineering we run agent-talk, a mailbox where agents in different projects ask each other questions but take no orders; we write about it in Labs.

FAQ

Can a multi-agent system run with no internet connection? Yes, if every layer is local: model server, orchestrator, MCP servers, logs and approvals. Turn off telemetry and tracing first.

How many agents should we start with? One agent plus code. Add a role only when a step needs different tools or permissions, and test each addition.

Does each agent need its own model? No. One served model covers most roles. Add a larger one only where your test set shows the small one fails.

What GPU do we need? It depends on model size, context per agent and how many run at once. An 8B model at 4-bit fits an 8 GB card for one agent at a time; parallel agents with long contexts need much more.

Is MCP required? No, but one MCP server per internal system can be reused by every agent and framework, with its rules in one place.

If you want agents working on your invoices, records or customer messages without the data leaving your building, we can look at the job and tell you how many agents it really needs. Book a consultation.