What is quantization?

Smaller numbers, smaller files

FP16 stores most values using 16 bits. Quantized formats reduce the precision of many weights and store additional information, such as scales, needed to reconstruct approximate values during inference. Mixed formats can preserve more precision in selected tensors.

That is why a “4-bit” model does not necessarily occupy exactly half a byte per parameter. The file format, architecture and quantization recipe affect the real size.

Choose a starting point, then test it

The planner uses effective averages of 4.8, 5.7 and 8.5 bits per weight for Q4, Q5 and Q8 respectively. These are disclosed planning coefficients, not exact sizes for every GGUF, GPTQ, AWQ or other artifact.

If the model fits comfortably, compare candidate files on your own tasks. A quantized model can make different mistakes, and the change depends on the original model and quantization method. No precision label guarantees a particular percentage of quality.

• Q4 is a compact planning starting point.
• Q5 gives weights a somewhat larger memory budget.
• Q8 trades more memory for higher numerical precision.
• FP16 is useful as an uncompressed comparison.

Weight precision and cache precision are separate

A Q4 model can still use an FP16 KV cache. Quantizing weights does not automatically shrink conversation memory. Some runtimes support quantized cache types, with compatibility and performance tradeoffs.

The planner defaults to FP16 cache. Enable Q8 cache only when your runtime and chosen model support it. Check the exact file size and observed memory use before relying on a tight estimate.