The short answer
Start a fresh chat with a short prompt, check that the model is using your GPU, and repeat the request after the model has loaded. Then try the same test with a smaller model that fits comfortably. This separates loading delays, long-prompt work and memory pressure.
Read the essentials below. Open Technical detail whenever you want to go deeper.Find the slow part
Time one ordinary request from pressing Send to receiving a useful answer. Watch whether the delay happens only on the first request, before every answer, or throughout generation. A single tokens-per-second number can hide most of that experience.
| What you notice | What to check first |
|---|---|
| First request is slow; the next is fast | Model loading and whether the app unloads it between uses. |
| Long pause, then fast output | Prompt length, prompt processing, queueing and hidden reasoning. |
| Words arrive slowly from the start | GPU placement, memory pressure and competing work. |
| Only long chats become slow | Growing input, cache memory and whether old input is being processed again. |
Technical detailRead Ollama timing fields correctly
The generate API reports load_duration, prompt_eval_duration and eval_duration separately, in nanoseconds. Divide eval_count by eval_duration / 1e9 to estimate generation tokens per second. That measures one phase, not the full wait.
For a hypothetical response with 240 generated tokens over 12,000,000,000 nanoseconds, generation is 20 tokens/s. If loading and prompt processing took another 18 seconds, the whole request still took about 30 seconds. Streaming time to first token needs its own client-side measurement.
Record a cold run and several warm runs separately. A repeated prompt may reuse cached work, so also test a different prompt of similar length. Reasoning output and visible answer tokens may be reported differently across tools.
Run a five-minute isolation test
Save your current settings so you can return to them. Change one thing at a time and keep the same test prompt.
- Pause other model jobs. Use one conversation and an explicit 4K or 8K context for this short test, within the model’s supported limit.
- Check GPU placement in the runtime. In Ollama, run ollama ps while the model is loaded; inspect PROCESSOR and CONTEXT.
- Send a short, concrete task, then repeat it once the model is warm. Save both timings.
- Try a smaller supported model with generous memory headroom. A large improvement makes the larger model’s placement or workload a useful lead.
- Restore your real prompt and context. If only this version is slow, focus on prompt size, conversation growth and application overhead.
Technical detailBenchmark the engine without confusing it with the chat app
llama-bench separates prompt processing (pp) from token generation (tg). Its synthetic workloads help isolate the engine, but they do not reproduce your full conversation, tool calls or document retrieval.
Record the exact artifact, engine build, GPU backend, offloaded layers, context, cache precision and concurrency. Compare like with like. A short synthetic generation test is insufficient evidence for performance late in a long chat.
llama-bench -m /path/to/model.gguf -p 512 -n 128 -r 3Make the next change match the evidence
If the model is repeatedly loading, consider keeping it resident while you work; remember that this keeps its memory occupied. If the model is split onto the CPU, test a smaller artifact or a shorter context. A split can be useful for capacity, but it is a different performance tradeoff.
If GPU placement looks right, record the runtime version and test its supported defaults before copying someone’s collection of tuning flags. Check sustained temperature, power and competing applications. Change batching, thread counts or speculative decoding only after you have a baseline.
A useful help request includes the command or settings, exact model file, hardware, prompt length, cold/warm timings and placement logs. Remove private prompts and credentials before sharing it.
From the community
LocalLLaMA users report long waits even on expensive GPUs, especially when a large model is split between GPU and system memory. Other threads report speed differences across apps. These are useful symptoms to investigate, not comparable benchmarks or proof that one app is always faster.
- LocalLLaMA: painfully slow inference on a 5090 (March 2026)
- LocalLLaMA: migrating from LM Studio to llama.cpp (August 2026)
These discussions informed the questions. Technical guidance is checked against the primary references; examples and checklists are editorial synthesis, not measured benchmarks.
See it on your own setup.
Change the model, hardware or context and see the memory budget update.
Try this example