CyberFreezeDev
Loading
Chunks of text a language model reads and writes, which set limits, speed and cost.
A token can be a word, part of a word or a symbol, and models split text differently. In English, 100 tokens is about 75 words.
A token is a unit of text from a model's vocabulary. It can be a whole word ("invoice"), part of a word ("recon" + "ciliation"), a space with a word, a number fragment or a punctuation mark. Each model family has its own tokenizer, so the same sentence can be a different number of tokens on different models.
OpenAI's rule of thumb for English is that one token is about four characters, or about three quarters of a word, so 100 tokens is roughly 75 words (OpenAI help). Treat that as an estimate only.
Before training, an algorithm learns the most useful frequent text chunks.
Before training, an algorithm such as byte-pair encoding scans a large text sample and learns the most useful frequent chunks, typically tens of thousands to a few hundred thousand of them.
Your text is broken into the longest matching chunks from that vocabulary.
Your text is broken into the longest matching chunks from that vocabulary.
Each token becomes a number ID, the only thing the model works with.
Each token becomes an integer ID. The model only ever works with these IDs.
The model outputs token IDs one at a time, which are turned back into text.
The model outputs token IDs one at a time, and the tokenizer turns them back into text.
Hosted models charge per input and output token, so long prompts at volume add up.
Hosted models charge per million input and output tokens, with output usually priced higher. Long prompts on high-volume tasks add up.
The context window is measured in tokens, not pages or words.
The context window is measured in tokens, not pages or words.
Tamil, Hindi and other scripts often use several times more tokens per word.
Tamil, Hindi and other scripts often take several times more tokens per word than English on many tokenizers, which raises cost and uses up context faster.
Most of the cost was a 2,000-word instruction block sent with every WhatsApp enquiry. Trimming it, RAG and prompt caching cut tokens without changing the answers.
A business sends every WhatsApp enquiry to an LLM with a 2,000-word instruction block attached. Most of the cost is that repeated block, not the customer's message. Cutting the instructions to what the task needs, moving reference material into RAG so it is only included when relevant, and using the provider's prompt caching where offered, reduces tokens per message without changing the answers.
Count with the model's own tokenizer on real inputs, not by words.
Count with the model's own tokenizer, such as OpenAI's tiktoken, on a sample of real inputs.
Verbose answers cost more and take longer. Ask for the length you need.
Verbose answers cost more and take longer. Ask for the format and length you need.
Counting characters or reversing strings is a weak spot. Do those in code.
Because models see tokens, not letters, tasks like counting characters or reversing strings are weak spots. Do them in code.
Yes. Multimodal models convert images and audio into tokens as well, and providers publish how they count them. A single image can cost hundreds or thousands of tokens depending on size.
Not per token. You pay for the hardware and power, so tokens matter for speed and for how many requests one GPU can serve rather than for a bill.
Last updated