CyberFreezeDev
Loading
Using a trained model on new input, such as each chat reply or document read.
Inference computes output from new input while the weights stay fixed. You pay per token to an API, or in hardware and power yourself.
Inference means feeding new input into a trained model and computing its output. No learning happens; the weights stay fixed.
For a large language model, inference produces text one token at a time. For a computer vision model it returns labels or boxes for an image. You pay for inference either per token to an API provider or in hardware, power and upkeep if you run models yourself.
The model weights go into GPU memory or system RAM.
The model weights are loaded into GPU memory (VRAM) or system RAM.
The whole prompt is processed in one pass; long prompts delay the first token.
The whole prompt is processed in one pass. Long prompts make this step slower, which shows up as time to first token.
One token at a time, with a KV cache so no step starts over.
The model generates one token, adds it to the sequence, and repeats. A KV cache stores earlier work so each step does not start over.
Each token is picked from the model's probabilities, shaped by temperature, until it stops.
Each next token is picked from the model's probabilities, shaped by settings like temperature, until a stop condition or length limit.
APIs charge per token; self-hosting costs hardware and power. Steady high volume favours owning.
API pricing is per input and output token; self-hosted cost is mostly hardware and electricity. High, steady volume often favours owning hardware.
A chat reply needs seconds; an overnight batch does not, so setups differ.
A WhatsApp assistant needs a reply in seconds; an overnight batch of thousands of invoices does not. The two need different setups.
With on-premise AI, inference runs on your machines and documents stay with you.
With on-premise AI, inference happens on your machines, so documents never cross to a third party.
A logistics firm reads delivery notes with a small local model by day for speed, and re-checks them overnight in batches on a larger one.
A logistics company reads delivery notes with an LLM. During the day, drivers upload photos and need an answer within seconds, so a small model handles them on a local GPU server with Ollama. At night, the full day's documents are re-checked in a batch with a larger model through vLLM, which groups many requests together to use the GPU efficiently. Same task, two inference setups, chosen by speed requirement.
Plan for peak users: one GPU fast for one person can be slow for twenty.
Plan for peak concurrent users. One GPU serving one user quickly can be slow for twenty at once.
Whole manuals in every request slow things and cost more. Retrieve only what is needed.
Pasting whole manuals into every request slows prefill and raises cost. Retrieve only what is needed with RAG.
Track tokens, latency and errors from day one, or costs drift unnoticed.
Track tokens per request, latency and errors from day one, or costs drift unnoticed.
It depends on model size, quantization and traffic. Small quantized models run on a single consumer GPU; larger models or many users need server cards with more memory. See GPU servers and AI hardware.
Per run, by a huge margin. But training is done once and inference runs for as long as the system is in use, so over a year inference is often the larger cost.
Last updated