On premise LLM for Indian businesses: models and hardware

An on premise LLM makes sense for an Indian business when the documents you want AI to read cannot leave your control and the work is steady enough to keep a GPU busy. The working recipe in October 2026 is an open-weight model whose licence you have read, served with vLLM or llama.cpp, on a GPU with enough memory for the weights plus the context your users need.

This guide is for the owner or IT head of a CA firm, clinic, manufacturer, logistics company or financial services business choosing between their own server and a cloud API. For the definition, see our glossary entry on on-premise AI. Here we cover the decisions: model, memory, hardware, software and who runs it.

This is general guidance, not legal advice. Check the DPDP Act, the Rules and your sector regulator's rules with your own adviser.

Why Indian firms keep the model in-house

The Digital Personal Data Protection Act 2023 makes your business, as the Data Fiduciary, responsible for personal data in its control, "including in respect of any processing undertaken by it or on its behalf by a Data Processor" (section 8(5)). A cloud AI provider that reads your prompts is one more Data Processor, which section 8(2) allows only under a valid contract. Section 16 lets the Central Government restrict transfers to countries it notifies.

The DPDP Rules 2025, published by MeitY on 13 November 2025, spell out the safeguards. Rule 6 asks for encryption or masking, access control, logs that can detect unauthorised access, backups, one year retention of those logs, and security clauses in each Data Processor contract. Rule 7 asks for a detailed breach report to the Data Protection Board within 72 hours. Both come into force eighteen months after publication, in May 2027. Rule 13(4) lets the government name personal data that a Significant Data Fiduciary must keep in India. Failing to take reasonable security safeguards can draw a penalty of up to ₹250 crore.

A model on your own server does not make you compliant. It shortens the list: no outside AI processor, no cross-border transfer for the AI step, and logs on a machine you control.

Step 1: choose the model by licence, then by task

"Open weights" is not one licence. Read the model card first. As of 26 October 2026:

Three rules for the shortlist:

1. Size mixture-of-experts models by total parameters. Mistral Small 4 computes with 6.5B parameters per token, but all 119B must sit in memory.
2. Test your languages. If staff write in Tamil or Hindi, try a model built for Indian languages, such as Sarvam 30B, beside a general one.
3. Score candidates on your own questions. Fifty to two hundred real questions from your documents beat any leaderboard.

Step 2: work out the GPU memory

The rule of thumb for weights: billions of parameters × bits per weight ÷ 8 = gigabytes. llama.cpp gives Q4_K_M as about 4.89 bits, Q8_0 about 8.50 and F16 16, so Llama 3.1 8B is 4.58 GiB at Q4_K_M and 14.96 GiB at F16. A 27B dense model such as Qwen3.8 27B works out to about 54 GB at 16-bit, 29 GB at 8-bit and 16.5 GB at Q4_K_M.

Then add the KV cache, which holds the conversation: 2 × layers × key/value heads × head size × bytes, per token. Llama 3.1 8B has 32 layers, 8 key/value heads and a head size of 128, so at 16-bit each token costs 128 KiB: 1 GiB for an 8,192 token chat, 4 GiB for a 32,768 token contract. Each concurrent user needs their own share, so ten people on long documents can need more memory for the cache than for 4-bit weights. Newer models can need less: Gemma 4 31B holds the full context in only 10 of its 60 layers, so read the model's config.json.

Add about 1 GB for the runtime and leave headroom. Our VRAM calculator adds the three parts and shows the working.

Step 3: pick a hardware tier

Memory and power are from NVIDIA's spec pages. "Roughly fits" is arithmetic from Step 2, not a benchmark.

What drives the cost:

• GPU memory. Your model and context choice sets the card class, and the card class sets most of the bill.
• Power and cooling. A 600 W card turns most of that into heat. Plan room cooling and a UPS for the whole machine.
• Buy, rent or private cloud. A rented GPU server or a cloud GPU in an Indian region keeps data in the country without buying hardware.
• People. Patching, monitoring and model updates take someone's time.

Step 4: choose the serving software

• vLLM for a shared server: PagedAttention for KV cache memory, batching across users, tensor parallelism across GPUs, FP8, GPTQ, AWQ and GGUF support, and an OpenAI-compatible API. SGLang is the other production engine the Qwen, DeepSeek and Mistral cards cover.
• llama.cpp and Ollama for one machine and a few users. llama.cpp runs GGUF files at 1.5 to 8 bits and can split a model between GPU and CPU when VRAM runs out, at lower speed. Ollama serves a subset of the OpenAI API at localhost:11434/v1.
• Not TGI for new builds. Hugging Face put it in maintenance mode, archived the repository on 21 March 2026 and points users to vLLM, SGLang and llama.cpp.

Because these speak the OpenAI API format, n8n or your own apps can point at your server instead of a public endpoint.

Step 5: plan who runs your on premise LLM

1. Sign-in and scopes. Staff use existing accounts; each team sees only its own documents.
2. Logs are personal data. Prompts carry client names and account details. Decide where logs live, who reads them and how long they stay; Rule 6 sets one year for security logs.
3. No route out. Internet access only for approved updates; telemetry and plugins that call home switched off.
4. Update with a test. Rerun your question set before swapping in a new model, and read any custom code a model repository asks you to run.
5. Monitoring and backups. Temperature, power, memory and response times; backups of configuration and document indexes.

When a cloud API is still better

If a few people use AI a few times a week, the GPU sits idle and a hosted API under a data processing contract is cheaper. A hybrid also works: private for client data, hosted for public text.

What we do here

Private AI deployment is one of the main things our engineers in Chennai build. In a private AI deployment we review the workload and data, score three to five open-weight models on an evaluation set from your documents, set out the buy, rent or private cloud choice, install and harden the stack behind your firewall, pilot it with one team for two to four weeks, and hand over a runbook. For the machine itself, see GPU servers and AI hardware builds. Our notes on what runs on one 8 GB laptop GPU are in local LLMs and private AI.

FAQ

What is the smallest on premise LLM setup worth trying? One GPU with 16 to 24 GB running a 7B to 14B model at 4-bit handles a pilot for a few users; gpt-oss-20b runs within 16 GB. Move up only when your test questions fail.

Can we use these models commercially? Apache 2.0 and MIT allow it. Llama and Muse Spark Open add attribution notices and thresholds, and some flagship models carry their own licences. Record the licence of every model you deploy.

Does an on-premise model make us DPDP compliant? No. It removes the outside AI processor and the transfer for that step. Notices, consent, retention, breach reporting and rights requests are still your processes; your legal adviser confirms the overall position.

How much GPU memory do we need for 20 users? Context per request matters more than headcount. Take the KV cache for a typical request, multiply by the people active at the same moment, then add the weights and about 1 GB.

GeForce or RTX PRO? The GeForce RTX 5090 has 32 GB. The RTX PRO Blackwell line reaches 96 GB on one card, lists ECC memory and has lower power versions. Either works for a pilot; for an unattended server, memory size and ECC usually decide.

If you are weighing an on premise LLM against a cloud API for client data, send us a sample of your documents and questions and we will tell you which model and hardware would handle them. Book a consultation.