Quantization stores model weights at lower numerical precision. Q4 generally needs less memory than Q8, while FP16 uses more. The exact file format adds overhead, and lower precision can change answers. A smaller file is useful only if it still does your task well.
Why fewer bits make smaller weights
A model stores many learned values. Using fewer bits for each value reduces their total size. Quantization represents values with less numerical detail, often using scales and groups of values to make that tradeoff useful.
Ignoring overhead, an imaginary eight-billion-parameter model uses about 4 billion bytes at four bits per value, 8 billion at eight bits, and 16 billion at sixteen bits. Divide by 1,073,741,824 to express bytes in GiB. Real formats also store metadata and may mix precisions, so this shortcut is a starting estimate.
Q4, Q8 and FP16 describe broad choices rather than identical packaging across every runtime. Check the exact file and its support. The calculator’s native option, when available, uses the recorded publisher weight format instead of assuming a generic conversion.
Smaller does not automatically mean better or worse for your task
Quantization can alter answers. Whether that matters depends on the model, conversion and task. A smaller file that fits comfortably may be more practical than a larger file that cannot run on your hardware, but neither choice establishes accuracy.
To compare meaningfully, keep the model, context, prompt and app settings fixed. Run several representative examples at each precision. Check the answers against the original material and note corrections needed. A single impressive response is a weak comparison.
Weight precision and KV cache precision are separate. Loading Q4 weights does not make the cache four-bit. Cache compression also requires runtime support, and some architectures use fixed native cache layouts.
Run a controlled calculator experiment
Make a prediction before changing the setting. You should expect the weights to change while the context-related memory remains the same.
- Choose a model with both Q4 and Q8 options. Keep the hardware and context fixed.
- If Loaded weights has an override in Advanced settings, clear it. A measured size override deliberately takes priority over the precision estimate.
- Compare Q4 and Q8. Record weights, total memory and headroom. The total does not necessarily double because its other components are unchanged.
- Use the new headroom to decide what to test in the app, then compare answer quality on the same tasks.
Put the idea to work
Keep the model, hardware and context unchanged. Switch weight precision from Q4 to Q8 and watch the model-weights part of the result grow.
Check your understanding
Does choosing Q4 also make conversation memory four-bit?
Reveal the explanation
No. Weight precision and cache precision are separate settings. A Q4 model can still use an FP16 cache. Change cache precision only when your runtime supports it.
Keep this in mind: Fewer bits usually save weight memory. Compare exact files and actual answers; do not confuse weight precision with cache precision.