Designing a Data Pipeline That Keeps Accelerators Busy

A data pipeline should be designed against the read rate a node can actually sustain, not against the size of the dataset. Most underperforming clusters fail here rather than in the model.

Start from the node

Establish what read rate keeps an eight-GPU node busy at the batch size and model you intend to run. That figure, multiplied by node count, is the requirement the storage tier has to meet. Sizing storage by capacity and hoping the throughput follows is the usual mistake and it produces exactly the symptom described elsewhere in this library: high reported GPU utilisation, low achieved throughput, no error anywhere.

Preprocessing

Preprocessing performed inside the training loop competes with training for the same resources. Where the transformation is deterministic, doing it once and storing the result is almost always the better trade, even at the cost of additional storage. Where it must be random per epoch, it belongs on CPU workers sized so they stay ahead of the accelerators rather than on the critical path.

Caching and prefetch

A working set that fits in a node-local cache removes most of the network from the steady state. Prefetch depth should be tuned so the next batch is resident before the current one completes; the correct depth depends on step time and read latency and is worth measuring rather than guessing.

Above roughly eight training nodes, concurrency rather than bandwidth becomes the limit, and a parallel filesystem that distributes metadata as well as data is generally the right answer. Below that, a well-specified NVMe-oF shelf is simpler and sufficient.

Measure the loader

Track data loader wait time as a first-class metric alongside step time. It is the fastest way to distinguish a slow model from a starved one, and the two look identical in a utilisation graph.