Tokens are pieces of text, not necessarily whole words. The context window holds instructions, history, documents and the answer being generated. The KV cache stores attention state for those tokens; its size depends on the model’s architecture and the runtime.
A token is not a word
A tokenizer breaks text into pieces the model represents as numbers. A piece can be a whole word, part of a word, punctuation or whitespace. Splitting varies by model, language and text. Code and unfamiliar words can use a different number of tokens than everyday prose.
On this site, 8K context means 8,192 tokens for one conversation. That budget includes instructions, history, supplied material and generated output. It is not an allowance of 8,192 words or an answer length setting.
Suppose you choose an 8,192-token budget. If instructions, history and a document consume 6,000 tokens, at most 2,192 remain for output before additional template overhead. The app may reserve output space or handle overflow differently. Check its actual token counts and settings.
Why context uses extra memory
While generating a response, a model can reuse attention information saved from earlier tokens. The KV cache stores that information so the runtime does not need to compute all of it again for each new token. It is working state, separate from the learned weights.
Full-attention cache generally grows with the tokens stored. Sliding-window layers can stop growing when old entries are discarded; some architectures also keep recurrent state. The runtime’s allocation strategy matters. This is why doubling context does not always double conversation memory for every model.
The calculator budgets using the selected architecture and context setting. An app may allocate the configured maximum up front or grow its cache while processing. An estimate at maximum context is therefore not necessarily the memory you observe at the beginning of an empty chat.
Supported context and useful context are different questions
A model’s supported limit does not guarantee that a workload fits your computer, runs quickly or uses every relevant detail accurately. An extended-context setting can also depend on particular runtime options.
Adding simultaneous requests creates more independent conversation state. For personal use, begin with one conversation. If you serve several people at once, test that actual concurrency and account for any runtime sharing behavior.
- Keep the model and weight precision fixed. Predict what changes when moving from 8K to 32K context.
- Compare conversation memory and weights. Weights should stay fixed; conversation memory follows the model’s cache layout.
- If the cache barely changes, inspect the model’s cache architecture and allocation assumptions rather than assuming the calculator is broken.
- In your app, try a short document with a known answer. Check both memory use and whether the answer cites the right passage.
Put the idea to work
Keep the model and precision fixed. Try 8K and 32K context. Watch conversation memory rather than weights: different architectures can grow at different rates.
Check your understanding
If a model supports a long context, is that memory free?
Reveal the explanation
No. Supported context is a model limit, not a hardware guarantee. Longer context can need more memory and processing time, and extended context may require special runtime settings.
Keep this in mind: Context includes input and output. More context can need more working memory; the model and runtime determine how much.