Model weights
We count all stored experts, embeddings and checkpoint tensors, including auxiliary or media weights where present. The count can exceed the rounded size in a model’s name. A text-only conversion or a file without speculative-decoding weights can be smaller.
Q4, Q5 and Q8 use planning averages of 4.8, 5.7 and 8.5 bits per weight; FP16/BF16 uses 16. These averages include typical scales and higher-precision tensors, but do not describe every quantization format. Enter measured loaded weight memory under Advanced to replace this estimate.
Native gpt-oss weights use the publisher’s tensor byte totals, which include MXFP4 expert matrices and higher-precision tensors. This assumes a runtime that keeps that format in memory. Dequantization or repacking can change the allocation.
Tensors with a required precision are counted separately. For example, Qwen3.8 Flash Next keeps its per-layer embedding table at 16 bits; selecting Q4 does not shrink that table. Each model’s details identify these exceptions.
Conversation state
The context includes instructions, input text and generated tokens. Each concurrent request gets its own full selected context. The calculator adds the model’s verified cache components:
- Full attention: layers × KV heads × (key dimension + value dimension) × tokens × bytes per element.
- Sliding attention: the same formula, capped at the sliding window for those layers when the runtime evicts old tokens. Advanced can switch to retaining the full context. Shared Gemma layers reuse existing caches; the two K and V tensors still count separately.
- Recurrent layers: fixed state per request. Gated DeltaNet uses value heads × key dimension × value dimension × 4 bytes, plus convolution channels × kernel width × 2 bytes per layer. Mamba uses its published state and channel dimensions in the same accounting.
- Compressed MLA: layers × tokens × (latent rank + positional dimension) × bytes per element, plus any verified index cache. The runtime must retain compressed latents.
FP16 cache uses 2 bytes per element; Q8 uses 1.0625 including block scales. Changing KV precision does not shrink FP32 recurrent state. Prefix sharing, cache offload and speculative decoding are not assumed.
Each model has an architecture-specific persistent-memory profile. The newest sparse models use a named runtime layout: native index buffers, pooling state and shared position tables are counted separately. These layouts use fixed native precisions and exclude speculative decoding and saved prefix states. Image, audio and video workspaces are additional even when their weights are included.
Long context
Options double from 2K through 1,024K tokens, with exact model limits added. Native and extended limits are stored separately. Extended context requires the publisher’s positional-scaling settings; changing this calculator does not configure your runtime.
Long prompt processing can allocate large temporary buffers. The default assumes memory-efficient attention and chunked prefill. Model support for a million tokens does not establish practical latency, quality at that length or sufficient temporary memory.
Hardware and GPU splitting
Each record specifies the chip or GPU bin, physical memory, device count and memory per device. Nominal memory is treated as binary DRAM capacity in GiB. Bandwidth is decimal GB/s and is kept per device; it is not multiplied into a generation-speed prediction. Missing published bandwidth stays blank.
Layer splitting uses a combined budget, with runtime buffers per device. Tensor splitting accounts for KV-head replication when heads cannot divide evenly, and full MLA cache replication on each rank. Recurrent state is conservatively replicated for tensor splitting. Separate model copies instead use each GPU’s budget independently; concurrent requests are per copy.
Every multi-GPU fit remains conditional. Some tensors are replicated, layers vary in size, and the output device can need extra memory. A combined fit is not a verified placement or a claim that every head split is supported.
Reserves and fit labels
Default runtime buffers per device are the larger of 1 GiB or 8% of that device’s ideal share of weights. System reserves are separate: generally 1 GiB per discrete GPU, and a documented configuration-specific allowance on unified-memory systems. Both are planning assumptions; replace them with observed values in Advanced.
- Fits with room: leaves at least 2 GiB or 10% of available memory, whichever is larger.
- Tight: within the budget but below that margin.
- Needs more memory: the complete estimate exceeds the budget.
- Possible fit: depends on splitting or a particular runtime cache/weight format.
- Unverified: conversation-state accounting is incomplete.
Calculations use unrounded inputs. Displayed numbers round to one decimal. Allocation padding, runtime workspaces and driver use can vary; fitting in memory does not predict speed.
Catalog coverage
53 models and 217 hardware configurations, checked 2026-09-06. Model configurations link to pinned publisher revisions. Newer similar-size releases replace older choices when the older version no longer has a distinct use. Relevant older hardware stays because installed and used machines still matter.
Announced hardware is listed separately and excluded from calculator defaults. Release status does not verify retailer stock. CPU-only memory layouts, heterogeneous CPU/GPU tiers and custom rack-scale systems need separate modeling.
Catalog 2026-09-06.2; calculator 2.0.0. Download the catalog.