GPU Cluster Scheduling: Resources, Gangs, and Priorities

Introduction

Welcome to our second lesson on GPU Datacenter Systems. In the previous lesson, we constructed the physical blueprint of a large-scale ML training cluster, detailing the compute, networking, and storage tiers. We saw how thousands of GPUs are interconnected with high-speed fabrics to form an immensely powerful, but also immensely expensive, computational instrument.

This raises a critical question: how do we efficiently and fairly manage access to this shared resource for dozens of teams and hundreds of jobs? This is where job schedulers come in. They are the master conductors of the cluster, deciding which job runs where and when.

This lesson directly addresses the learning outcome: to explain how job schedulers like Slurm and Kubernetes manage resource allocation, gang scheduling, and priorities in a multi-tenant GPU cluster. We will explore the two dominant philosophies in cluster scheduling—the High-Performance Computing (HPC) approach and the Cloud-Native approach—and see how they are embodied in Slurm and Kubernetes, respectively.

1. The Core Challenge: Scheduling for Deep Learning

Scheduling workloads on a multi-tenant GPU cluster presents unique challenges that differ significantly from traditional CPU-based cluster computing. These challenges stem from the nature of the hardware and the structure of distributed AI jobs.

Let's begin by reading a short introduction from a Microsoft Research paper that analyzed their large-scale production GPU clusters. It sets the stage perfectly.

The reading highlights two critical constraints:

1. Monolithic Resources: Unlike CPUs, which can be finely time-sliced to serve many processes, a GPU is generally a monolithic resource. A single training job typically occupies an entire GPU, making it difficult to share at a fine granularity.
2. Gang Scheduling: Distributed training jobs require multiple GPUs to be available at the same time. The scheduler cannot start part of the job now and the rest later; it must allocate all the requested resources simultaneously. This "all-or-nothing" approach is known as gang scheduling.

These constraints lead to a significant secondary problem: resource fragmentation.

The Microsoft paper you read earlier provides excellent terminology for the consequences of these challenges. It divides the time a job spends waiting in a queue into two categories:

• Fair-share delay: This occurs when a user or group has already used up their allocated resource quota. The job must wait for other jobs from the same group to finish, ensuring fairness across tenants. This is a policy-driven delay.
• Fragmentation delay: This occurs when a job is within its quota, but the scheduler cannot find enough co-located resources to satisfy the gang scheduling and locality requirements due to fragmentation. As the paper notes, this accounts for the vast majority of queuing time for large jobs.

Effectively managing quotas, priorities, and fragmentation is the central task of a job scheduler. Today, two major systems, Slurm and Kubernetes, approach this task from fundamentally different philosophical standpoints.

2. Two Philosophies: HPC vs. Cloud-Native

The design of a scheduler is deeply influenced by its core assumptions about the relationship between resources and workloads. The "Slinky" presentation from CNCF provides a brilliant articulation of these two opposing worldviews.

Let's summarize the key points from the video. This distinction is crucial for understanding everything that follows.

Dimension	HPC (e.g., Slurm)	Cloud-Native (e.g., Kubernetes)
Core Assumption	Finite Resources, Infinite Workload	Infinite (Elastic) Resources, Finite Workload
Primary Goal	Maximize throughput and utilization of fixed, expensive hardware.	Ensure availability and resilience of services.
Workload Model	Batch jobs that run to completion.	Long-running, stateless services (microservices).
Scheduling Mechanism	Queues & Priorities: Jobs wait their turn based on priority, fair-share, etc.	Reconciliation & Scaling: Continuously tries to match the current state to a desired state. Scales horizontally by adding more resources.
Key Feature	Backfill Scheduling: Improves utilization by running small jobs in idle gaps.	Self-healing: Automatically restarts failed components on healthy nodes.

With this theoretical framework in mind, let's dive into how each system works in practice.

3. Slurm: The HPC Workload Manager

Slurm (Simple Linux Utility for Resource Management) is the de facto standard scheduler in the HPC world and manages the majority of the world's supercomputers. Its design is a direct reflection of the HPC philosophy.

Architecture and Workflow

• Architecture: Slurm has a centralized architecture. A primary controller daemon (slurmctld) holds the state of all jobs and nodes in memory, allowing for extremely fast scheduling decisions. This is paired with a slurmd daemon on each compute node.
• Workflow: The model is imperative. A user submits a batch script (e.g., a shell script) using the sbatch command. The script contains directives specifying the required resources (e.g., number of nodes, GPUs per node). The job is then placed in a queue.

Resource Allocation and Priorities

Slurm is built around the concepts of queues (called partitions) and priorities.

• Allocation: When a job is submitted, Slurm places it in a partition. Each partition can have its own set of policies, limits, and priorities. When a job reaches the front of the queue and its required resources are available, Slurm grants it exclusive access to those resources for the duration of the job. GPUs are requested as a "Generic Resource" (GRES).
• Priorities & Fairness: Slurm uses a sophisticated, pluggable priority system. A common policy is Fair Share, which gives each user or group a target share of the cluster. A user who has recently run many jobs will have their priority lowered, while a user who has been idle will have their priority raised. This prevents any single user from monopolizing the cluster. Preemption is also supported, where a high-priority job can suspend or cancel a lower-priority job to claim its resources.

Gang Scheduling

Gang scheduling is a native, core concept in Slurm. When a user requests a multi-node, multi-GPU job, Slurm understands that it is an atomic unit. It will not start any part of the job until all requested nodes and GPUs are secured. This avoids the partial allocation deadlocks that can plague other systems.

A key feature that complements this is backfill scheduling. While a large, high-priority job is waiting for resources to free up, the scheduler identifies idle resources and "backfills" them with smaller, shorter jobs that are guaranteed to finish before the large job is scheduled to start. This dramatically increases overall cluster utilization.

For a concise overview of Slurm and Kubernetes' different approaches, you can refer to the resource Slurm vs Kubernetes: How HPC and Cloud-Native Orchestration .... Its section on "Scheduling Model and Policies" provides an excellent summary.

4. Kubernetes: The Cloud-Native Orchestrator

Kubernetes was designed at Google to orchestrate microservices at scale. Its application to batch workloads like ML training is a more recent development, which explains some of its different design choices.

Architecture and Workflow

• Architecture: Kubernetes has a distributed, highly available control plane. Its state is stored in a distributed key-value store (etcd). A scheduler component watches for newly created workload objects and assigns them to nodes. This design prioritizes resilience over raw scheduling speed.
• Workflow: The model is declarative. A user defines the desired state of their application in a YAML file (e.g., "I want this container image to be running"). They submit this to the Kubernetes API server, and the system's controllers work to make the cluster's actual state match the desired state. The fundamental unit of scheduling is a Pod, which is a group of one or more containers.

Resource Allocation and Priorities

• Allocation: Kubernetes uses a requests and limits model. A Pod requests a certain amount of CPU and memory, and the scheduler finds a node that has sufficient available capacity. It does not typically grant exclusive access to nodes; multiple Pods from different jobs are often "bin-packed" onto the same node. GPUs are handled via a device plugin framework, advertised as a special resource like nvidia.com/gpu.
• Priorities & Fairness: Fairness is typically managed via ResourceQuotas applied to Namespaces (a way to partition a cluster for different teams/projects). This sets a hard limit on the total resources a group can consume. PriorityClasses can be assigned to Pods, allowing higher-priority Pods to preempt lower-priority ones if resources are scarce.

Gang Scheduling

This is the most significant difference from Slurm: gang scheduling is not a native feature of the default Kubernetes scheduler. The default scheduler evaluates one Pod at a time. For a distributed training job consisting of 16 Pods, it might successfully schedule 8 of them, but if the cluster runs out of resources, the remaining 8 will get stuck in a pending state. The 8 running Pods will wait forever for their peers, leading to a deadlock that wastes valuable GPU resources.

To solve this, the community has developed several solutions that extend Kubernetes:

• Custom Schedulers: Projects like Volcano and Kueue are complete scheduling systems designed for batch workloads that run within Kubernetes. They introduce the concepts of jobs and queues, providing robust gang scheduling and other HPC-like features.
• Scheduler Plugins: The Kubernetes scheduling framework is extensible. Simpler forms of gang scheduling can be implemented with plugins that ensure a group of Pods (often defined by a PodGroup or JobSet object) are scheduled all-or-nothing.

The following video provides an excellent deep-dive into a project that aims to bring Slurm's powerful scheduling logic directly into a Kubernetes environment, highlighting these very challenges.

Conclusion

In this lesson, we dissected the complex task of scheduling jobs in a multi-tenant GPU cluster. We saw how the unique requirements of deep learning—monolithic GPU resources and the need for gang scheduling—create significant challenges like resource fragmentation.

Key Takeaways:

• Job schedulers are critical for managing the allocation of expensive and scarce GPU resources efficiently and fairly among multiple users (multi-tenancy).
• Slurm, representing the HPC philosophy, assumes finite resources and optimizes for throughput. It uses a centralized controller, job queues, and native gang scheduling and backfill to maximize hardware utilization. Job priorities are managed through mechanisms like Fair Share.
• Kubernetes, representing the cloud-native philosophy, assumes elastic resources and optimizes for service availability. It uses a declarative model with a distributed control plane. It excels at bin-packing diverse workloads but lacks native gang scheduling, requiring extensions like Volcano or Kueue for robust batch job management.
• The fundamental difference in handling gang scheduling is a key trade-off: it is a core, highly-optimized feature in Slurm, while it is an add-on in the more flexible, extensible Kubernetes ecosystem.

Preview of the Next Lesson:

Our discussion of Kubernetes highlighted the central role of containers. In our next lesson, we will focus on this critical technology. We will explore how containerization (e.g., Docker, Apptainer) is used to manage dependencies and ensure reproducibility in GPU environments, a practice that is now fundamental to both Kubernetes-based and modern Slurm-based AI workflows.