Kubernetes GPU
Kubernetes GPU farm: multi-tenant scheduling done right
Sharing expensive GPUs across teams without chaos requires more than nvidia.com/gpu limits. GPU Operator, Kueue, MIG and quotas are the real stack.
Written by
YAPIO
Published on
Jul 17, 2026
Why Kubernetes for GPUs at all
Slurm remains the reference for large synchronous training. But most enterprises run a mixed reality: several teams, inference services, notebooks, batch fine-tuning, CI jobs. Kubernetes wins there because it speaks the language your platform team already operates: namespaces, RBAC, GitOps, observability. The GPU Operator handles drivers, the device plugin, DCGM metrics and node feature discovery declaratively, so a GPU node joins the cluster in minutes instead of a hand-tuned afternoon.
The trap is treating GPUs like CPUs. A GPU is not fractional by default, jobs are often gang-scheduled, and idle allocation burns money at a rate that makes CPU waste look trivial.
Kueue, gang scheduling and fair sharing
Distributed training needs all-or-nothing placement: launching 7 of 8 workers deadlocks the fabric and wastes every allocated GPU. Kueue brings queueing semantics to Kubernetes: ClusterQueues with quotas per team, cohorts for borrowing idle capacity, and job admission only when the full gang fits. Combined with priority classes and preemption, a research team can borrow the inference team’s idle nights without anyone filing a ticket.
For sharing a single card, choose deliberately: MIG partitions an H100 or A100 into hardware-isolated slices with predictable performance, ideal for inference tiers. Time-slicing is softer, fine for notebooks and dev, wrong for latency-sensitive serving. Document which profile each queue exposes and enforce it in the platform, not in tribal knowledge.
Tenancy, quotas and the paved road
Multi-tenant GPU clusters fail through leaks, not through scheduling. Isolate teams with namespaces, NetworkPolicies and separate storage classes; expose golden Helm charts or an internal template for training and serving jobs so nobody reinvents pod specs with privileged mode. Chargeback matters even internally: publish per-team GPU-hour dashboards from DCGM data, because visible cost is the only durable incentive against squatting.
Run the whole thing through GitOps. Queue definitions, quotas, MIG profiles and driver versions belong in a repository with review, so capacity policy changes are auditable decisions instead of kubectl surprises.