Slurm

A widely used HPC job scheduler (sbatch, squeue, fair-share, preemption). Researchers often prefer its UX even when the cluster underneath runs on Kubernetes.

How it works

Slurm (Simple Linux Utility for Resource Management) manages job queues on HPC clusters. Users submit jobs with resource requests (number of GPUs, memory, time limit) via sbatch scripts. Slurm schedules these jobs across available nodes according to priority (based on fair-share, partition, and QOS policies), tracks usage, and handles preemption of lower-priority jobs when resources are needed. It supports MPI for distributed training and integrates with cluster monitoring and accounting systems.

Why it matters

Slurm is the dominant job scheduler in academic AI research and many enterprise ML clusters. Understanding Slurm is essential for any researcher running experiments on university or national lab clusters. Its fair-share scheduling, preemption support, and accounting features make it well-suited to multi-user research environments where compute is shared and fair access matters. Many frontier model training runs — particularly academic ones — are orchestrated via Slurm even when the underlying infrastructure has evolved beyond traditional HPC.