The short answer
Compare two models that fit with your real context, then test the tasks you actually do. A larger Q4 model is worth trying, but a smaller Q8 model may be the better tool if it is more reliable on your task, leaves room for context, or responds much sooner.
Read the essentials below. Open Technical detail whenever you want to go deeper.Choose two realistic candidates
Use an instruct or chat model for ordinary conversation. Check that your runtime supports the exact architecture and artifact. Compare within one model family when you want to study size versus precision; compare across families when your goal is simply to find the most useful tool.
Quantization methods differ in memory use, supported hardware and how they approximate the original model. Two files labeled four-bit need not have identical sizes or quality. Start with actual download sizes, then budget the cache and runtime separately.
Technical detailWhy equal file sizes do not imply equal capability
Using this site’s planning coefficients, a hypothetical 8B model at 8.5 bits per weight needs about 7.92 GiB for weights. A 14B model at 4.8 bits needs about 7.82 GiB. That is roughly equal weight storage, not equal total memory or performance.
AWQ is an example of why the quantization recipe matters: it uses activation information to identify salient weights and improve low-bit approximation. Results from one method, model and evaluation set are not a guarantee for every file with a similar precision label.
For mixture-of-experts models, distinguish total stored parameters from the subset activated per token. The original Qwen3-30B-A3B card lists 30.5B total and 3.3B activated. A small active count does not turn the complete weights into a 3.3B-size download.
Make a small test set before seeing the answers
Choose ten representative prompts and write down what success means. Ten is a manageable starting exercise, not a statistically conclusive benchmark. Include a few difficult examples that have caused you trouble before.
| Work you do | Example checks |
|---|---|
| Three everyday tasks | Useful answer, correct facts and instructions followed. |
| Three exact tasks | Extract a number, produce valid JSON, or pass a small code test. |
| Two longer inputs | Find a specific fact and combine evidence from separate passages. |
| Two missing-information cases | Identify what cannot be established instead of inventing an answer. |
Technical detailScore useful work, not impressive-looking prose
For each prompt, record pass/fail on the must-have requirement, a short error note, and total time to a usable result. Separate factual correctness, formatting and style. A valid JSON object with the wrong amount is still wrong.
Hide the model labels when comparing prose. Repeat close or inconsistent cases, and preserve the first attempts instead of selecting the nicest answer. For code, run an appropriate test in an isolated project; confident explanations cannot substitute for correct behavior.
Perplexity measures predictive performance on a text sequence and is affected by tokenization and evaluation setup. It is useful research evidence, but it does not directly report whether your spreadsheet extraction or code change succeeds.
Keep the comparison fair enough to be useful
Use the same input, context allowance and output budget. Start each trial in a fresh chat. Record each model’s documented template, sampling settings and thinking mode; do not force an unsupported setting merely to make the numbers look identical.
Warm each model before measuring normal response time, and keep the cold-start result separately. If one candidate offloads to the CPU or needs a much smaller context, include that in the decision. It changes the experience you are evaluating.
Choose the candidate that meets your minimum quality with acceptable waiting time and memory headroom. If they trade wins across tasks, keep both for different jobs. When an apparent quality gap disappears after fixing a template or truncation setting, rerun the comparison before buying hardware.
From the community
A June 2026 LocalLLaMA discussion asks whether small Q8, medium Q4 or large Q2 is optimal at a similar memory budget. The conflicting answers are a useful reminder that there is no universal winner. This guide turns that argument into a repeatable comparison.
These discussions informed the questions. Technical guidance is checked against the primary references; examples and checklists are editorial synthesis, not measured benchmarks.
See it on your own setup.
Change the model, hardware or context and see the memory budget update.
Try this example