CyberFreezeDevLong documents
A 200-page contract may fit a large model but need splitting on a small one.
Read more about Long documents
A 200-page contract may fit in one request on a large model, or need splitting on a small local one.
- Home
- AI glossary
- Context window
What is a context window?
The most text a model can handle in one request, your input and its reply together.
The plain definition
Input and output share one token limit. About 100 tokens make 75 English words, so 128,000 tokens hold roughly 96,000 words.
Read more about The plain definition
The context window is the limit on how many tokens a model can process at once. It covers input and output together. Current models range from a few thousand tokens in small local models to hundreds of thousands, and some hosted models accept a million or more.
As a rough guide from OpenAI's help article on tokens, 100 tokens is about 75 English words, so a 128,000-token window holds roughly 96,000 English words, about the length of a long novel. Tables, code and many Indian languages use more tokens per word.
How it works in a real request
-
01 System instructions
Rules about role, tone and format use some tokens on every call.
Read more about System instructions
Rules about role, tone and format take some tokens on every call.
-
02 Retrieved or attached content
Documents, search results and tool outputs are added next.
Read more about Retrieved or attached content
Documents, search results and tool outputs are added.
-
03 Conversation history
Earlier turns are resent every time, so long chats keep growing.
Read more about Conversation history
Earlier turns are resent each time, so long chats keep growing.
-
04 Room for the answer
What is left caps the reply. Over the limit, the app must trim or summarise.
Read more about Room for the answer
Whatever remains is the most the model can write back. If the total exceeds the window, the app must cut or summarise something.
Where it matters in business
Cost and speed
Every token is processed on every call, so bigger prompts cost more and run slower.
Read more about Cost and speed
Every token in the window is processed on every call. Bigger prompts mean higher API bills and slower inference.
Local hardware
Longer context needs more VRAM for the KV cache, on top of the model weights.
Read more about Local hardware
On your own GPU, a longer context needs more VRAM for the KV cache, on top of the model weights.
A practical example
A support bot lost order numbers in long chats because old messages were trimmed. Keeping a short summary of key facts at the top fixed it.
Read more about A practical example
A support assistant for an online store kept forgetting the order number mentioned at the start of long WhatsApp chats. The cause was the app trimming old messages to stay under the window. The fix was to pull key facts (order number, customer name, issue) into a short structured summary kept at the top of every request, and to drop small talk instead of facts.
Common mistakes
Assuming big window means good recall
Models use the start and end of a long context better than the middle.
Read more about Assuming big window means good recall
Research such as "Lost in the Middle" (Liu and colleagues, 2023) found models use information at the start and end of a long context better than the middle.
Stuffing instead of retrieving
Sending everything is slower and often less accurate than retrieving the right passages.
Read more about Stuffing instead of retrieving
Sending everything is slower and often less accurate than sending the few right passages with RAG.
Forgetting the output budget
A long prompt can leave too little room, cutting the answer off mid-sentence.
Read more about Forgetting the output budget
A long prompt can leave too little room, and the answer gets cut off mid-sentence.
Questions
Does a bigger context window mean the model remembers past chats?
No. The window only covers the current request. Any memory across sessions is built by the application, which stores notes and sends them back in.
How do we count tokens before sending?
Use the tokenizer for your specific model, such as OpenAI's tiktoken library or the tokenizer that ships with an open-weight model. Counts differ between models.
Last updated