Rack-Scale Inference Platform

Inference is a different purchase from training, and buying it like training is the common way to overspend. A training node is sized for a job that ends. A serving box runs every hour of every month, so the figures that decide it are throughput per watt and how much of the machine is actually in use at the median hour, not peak FLOPs.

That makes right-sizing the whole exercise. The CG290 is a single-socket 2U taking up to four accelerators, which is the correct machine for serving one model at moderate concurrency and the wrong one to grow into. The CG480 is 4U with up to thirteen PCIe 5.0 x16 slots and up to eight accelerators, which is the machine for consolidating several models onto one box or for a serving tier that is going to grow. Start from the model and the concurrency, then pick the chassis.

These are MGX systems, so the accelerator is a decision rather than a fixture. H200 NVL where memory capacity per card decides how large a model fits; RTX PRO 6000 Blackwell Server Edition where throughput per ringgit matters more than capacity. PCIe rather than SXM is the trade being made: less inter-GPU bandwidth than an SXM node, at a fraction of the power and facility burden, which for serving is usually the right side of the trade.

Two details that decide the build. Storage sits on the chassis choice: the Intel CG480 carries twenty front E1.S NVMe bays, the EPYC version carries eight U.2 with twelve memory channels per socket, and which matters depends on whether the working set lives on the box or on the network. And the CG481 is the CG480 with eight 400G QSFP ports on ConnectX-8 built in, at the cost of four expansion slots. That is worth it when the deployment is multi-node from day one, and not worth it for a single box.