The Economics of Serving Large Models

Cost per token is determined by batch size, memory bandwidth and interconnect together. Optimising any one in isolation usually moves the cost somewhere else.

Batch size

Larger batches raise throughput and lower cost per token, up to the point where they raise latency past what the service can accept. That ceiling is a product decision rather than an engineering one, and it should be established before the hardware is specified, because it determines how much of the accelerator can actually be used.

A serving tier sized without a latency target tends to be specified for peak throughput and then run at batch sizes far below what it was built for, which is an expensive way to discover the requirement.

Memory bandwidth

Serving large models is bandwidth-bound far more often than it is compute-bound. Weights have to be read for every forward pass, so bandwidth sets the ceiling on tokens per second regardless of available FLOPS. This is why H200 class hardware serves competitively against newer accelerators: the capability that separates them is training capability, and a serving tier does not exercise it.

Interconnect

Interconnect matters only when a model does not fit in one node. Where it does fit, an expensive fabric buys nothing for serving. Where it does not, the fabric becomes part of every single request rather than an occasional collective, and it dominates latency.

The practical consequence is that model placement is an economic decision. Keeping a model within a node, by quantisation or by choosing a serving-appropriate variant, is frequently worth more than any hardware upgrade.

Utilisation, again

A serving fleet provisioned for peak and idle most of the time pays for capacity it does not use. Cost per token at average load, not peak, is the number that should drive the sizing.