Kubernetes

An open-source system for running and healing containerised workloads across a cluster — the usual backbone for modern cloud training and serving platforms.

How it works

Kubernetes orchestrates containerised applications across a cluster of machines. Workloads are described as declarative configurations (pods, deployments, services). The Kubernetes control plane schedules pods to nodes based on resource requests, monitors their health, and automatically restarts failed containers. For ML serving, Kubernetes manages the deployment of model inference containers, load balancing across replicas, auto-scaling based on request traffic, and GPU resource allocation.

Why it matters

Kubernetes is the standard deployment platform for production AI services. Nearly every cloud-based LLM API is served on Kubernetes clusters. Its declarative approach, automatic healing, and auto-scaling make it ideal for managing the unpredictable load patterns of AI services. For ML practitioners, understanding Kubernetes is increasingly expected for deploying models at scale — from staging individual model endpoints to managing the complex multi-service pipelines of modern RAG and agentic AI systems.