Scheduling a Shared Cluster Without Starving Anyone

Sharing a cluster across teams without starving anyone is largely a queue design problem, and the highest-return intervention is usually not a scheduler change at all.

Move development off the cluster

A large share of what queues on a shared cluster never needed rack-scale hardware. Exploratory work, debugging, environment setup and small fine-tuning runs occupy allocation slots and delay the production runs that genuinely require the cluster. Moving that work to desk-side systems raises effective throughput more than any scheduling policy, because it removes the demand rather than reordering it.

It also improves the experience on both sides. Researchers stop waiting four hours for an allocation to run something that takes twenty minutes, and the cluster queue starts reflecting real production demand.

Fair share, with a preemption policy

For what remains, fair-share allocation across teams with a defined preemption policy works better than strict priority ordering, which tends to let one well-connected team consume the cluster indefinitely. Preemption needs checkpointing to be viable, which is another reason to keep checkpoint retention generous and the storage tier for it cheap.

Publish the numbers

Queue wait time by team, and utilisation by team, should be visible to everyone using the cluster. Most contention disputes resolve themselves once the data is shared, and the ones that do not are at least arguing about the same facts.