The short answer
Reduce the configured context and run one conversation first. Leave room for the answer, close other loaded models, and inspect actual memory use. Then increase context in steps. A model’s advertised maximum is a supported limit, not a promise that it fits your GPU.
Read the essentials below. Open Technical detail whenever you want to go deeper.Budget the complete conversation
System instructions, earlier messages, retrieved passages, tool results and new output all compete for context. A coding assistant can send substantial input even when your typed request is one sentence.
Start with the runtime’s actual token count. As a planning example, an 8,192-token window with 1,200 tokens of instructions, 3,500 of history and 1,000 of documents leaves 2,492 for the answer and other formatting. Those are illustrative counts; measure your own.
Technical detailWork through the cache arithmetic
For ordinary full-attention layers, cache bytes per conversation are approximately 2 × layers × KV heads × head dimension × tokens × bytes per value. The first 2 accounts for keys and values. Use KV heads, which can differ from query heads.
A hypothetical model with 32 layers, 8 KV heads and a head dimension of 128 uses 128 KiB per token with two-byte values: 1 GiB at 8,192 tokens, or 4 GiB at 32,768. Four independent full-length conversations would need 16 GiB at the latter length, before weights and other allocations.
This example assumes equal key/value dimensions and no sharing or compression. Sliding-window, recurrent, hybrid and latent-attention models need different accounting. Use the model’s architecture and observed runtime allocations rather than applying this formula universally.
Change the cheapest things first
Preserve important facts before shortening a chat. Keep a brief working summary plus the original files you still need, and start a new conversation. Summaries can omit details, so retain references for anything you must verify.
- Use one active request and unload models you are not using.
- Set an explicit context that accommodates the current task, then reload if your runtime applies this at load time.
- Remove duplicate documents, repeated instructions and irrelevant tool output. Keep the evidence needed for the question.
- Increase context gradually while recording peak memory and response latency. Test a genuinely long prompt, not just an empty large window.
- If it still does not fit, compare a smaller model or supported cache quantization. Retest answer quality after each change.
Technical detailWeight quantization, cache quantization and Flash Attention
Q4 in a model filename usually describes weight storage. It does not mean the cache is also four-bit. Cache precision is a separate setting, with model and backend restrictions.
In Ollama, quantized cache requires Flash Attention; its documented cache options include f16, q8_0 and q4_0. The cache setting is global. Check the running version and test a representative long task after changing it, especially exact extraction or code.
Flash Attention reduces intermediate attention memory traffic and storage; it does not eliminate the need to retain conversation state. Smaller cache values save memory, but metadata and other allocations mean total process memory will not fall by the same ratio.
Distinguish a memory failure from a context failure
An allocation error at load time can mean the configured cache is reserved up front. A failure later can involve a growing cache or temporary buffers. A request rejected for too many tokens is a context-limit error; a fluent answer that forgot an earlier fact is a quality symptom. Keep the exact error and timing.
Inspect the logged allocation after changing context. Some caches grow dynamically; others reserve capacity in advance. The configured limit, currently occupied tokens and allocated memory are separate numbers.
For a document collection, retrieve relevant passages instead of sending the whole library every turn. For a task that truly needs many distant passages at once, test both memory fit and whether the model reliably uses those passages.
From the community
A LocalLLM question describes the familiar cliff: short context fits, longer context spills into RAM and becomes slow. A separate LocalLLaMA cache complaint was later updated after a runtime and artifact fix. Both suggest checking allocation and software versions before concluding that more hardware is the only answer.
- LocalLLM: more context without losing speed (August 2026)
- LocalLLaMA: cache memory issue, with an author’s correction (April 2026)
These discussions informed the questions. Technical guidance is checked against the primary references; examples and checklists are editorial synthesis, not measured benchmarks.
See it on your own setup.
Change the model, hardware or context and see the memory budget update.
Try this example