Sizing Storage So Your GPUs Are Never Waiting

Storage for a training cluster is routinely specified as a capacity number: how many petabytes of training data need to be held. That is the easy half of the question and not the half that determines whether the cluster performs.

The number that matters

What matters is sustained read throughput against the access pattern of the training job, measured against how fast the accelerators consume data. A cluster of eight-way nodes can consume data faster than most enterprise storage tiers deliver it, and when it does, expensive hardware sits idle waiting.

This shows up as low GPU utilisation with no obvious fault. Everything is running. Everything is healthy. The cluster is delivering a fraction of what it cost.

Tiering, honestly

A hot NVMe-oF tier sized to hold the working set of the current job, at throughput matching the node count. A capacity tier on spinning disk for raw corpora and checkpoint archives that do not justify flash.

Where many nodes read the same dataset concurrently, a parallel filesystem is worth the operational overhead it introduces. Where they do not, it is overhead without benefit, and a simpler design performs as well.

Checkpoints are the part people underestimate

Checkpoint writes are bursty, large, and happen at the worst possible moment for a job already under memory pressure. Size the write path for the checkpoint burst rather than the average, and confirm the network path can carry it without contending with the training traffic.

A practical starting point

Measure the read throughput of one representative training job against one node. Multiply by node count, add checkpoint burst headroom, and specify from that. It is a considerably more useful number than a petabyte target and it takes an afternoon to establish.