Monitoring GPU Utilisation That Actually Means Something

GPU utilisation percentage is a poor signal on its own, and clusters are routinely reported as busy while doing very little useful work.

The number counts whether a kernel is resident, not whether it is accomplishing anything. A GPU spinning on a memory-bound operation, or waiting on a collective, or processing a batch far too small to fill it, all report high utilisation. A cluster that feels slow while showing ninety percent utilisation is the normal case, not an anomaly.

What to measure instead

Achieved occupancy and memory bandwidth utilisation say whether the accelerator is doing work proportionate to its capability. Collective time as a fraction of step time says whether the fabric is the constraint. Data loader wait time says whether storage is. Between them these three usually identify which part of the system is setting the pace.

Step time is the number that matters to the people paying for the cluster, and it is worth tracking as the headline figure with the others as diagnostics beneath it.

The usual answer

When a cluster feels slow, the cause is most often the data path rather than the accelerators. A storage tier that cannot sustain the read rate a node requires produces exactly the pattern described above: high reported utilisation, low achieved throughput, and no error anywhere. Check the data loader before optimising anything on the GPU.