What to plan for
- GPUs
- Two supported 24 GiB GPUs; check the runtime’s model-splitting support.
- Platform
- A motherboard with appropriate slot spacing and PCIe connectivity.
- Memory & storage
- Plan enough system RAM to load and stage the model; 128 GiB is a useful capacity to evaluate.
- Power & cooling
- Two RTX 3090 cards can account for 700 W of board power alone. Engineer for the entire system.
Your first run
- Estimate aggregate memory, then verify that individual layers and buffers fit on each device.
- Configure the actual split explicitly and inspect device-level allocations.
- Test sustained inference and compare against a single higher-memory alternative.
The tradeoff
This is a configuration concept, not a compatibility-certified parts list. More components, heat and runtime complexity may outweigh an apparent used-GPU saving.
These are component selection criteria, not a verified bill of materials. Check compatibility, current prices, power and cooling before purchasing. The examples assume Q4 weight precision, 8K context, one request and FP16 KV cache.