Skip to content

Kubernetes

Kubernetes (k8s) orchestrates containers: it schedules them, restarts them when they die, scales them up and down, and routes traffic to them. For ML teams it is the layer that runs model services, batch jobs, and data pipelines reliably at scale.

Table of Contents

Installation / Setup

You usually don't install Kubernetes itself — you use a cluster provided by a cloud (EKS, AKS, GKE), a local one (kind, minikube, k3s), or a managed distribution. What you always install is kubectl, the CLI:

# macOS (Homebrew)
brew install kubectl

# Windows (Chocolatey)
choco install kubernetes-cli

# Verify
kubectl version --client
kubectl cluster-info   # talks to the cluster your kubeconfig points at

Kubeconfig (default ~/.kube/config) holds cluster addresses and credentials. kubectl config manages contexts (cluster + user + namespace triples):

kubectl config get-contexts            # list contexts
kubectl config use-context <name>      # switch cluster
kubectl config current-context         # where am I?

Core Concepts

Concept What it is Analogy
Pod The smallest unit: one or more containers sharing a network namespace A running process (or small group)
Deployment Declares a desired state of pods (image, replicas) and keeps it true The "desired state" manager
Service Stable network endpoint (DNS name + IP) in front of pods A load balancer / phone number
Namespace A partition of the cluster for isolating groups of resources A folder / team boundary
ConfigMap Non-secret configuration as key-value pairs Config file / env vars
Secret Sensitive configuration (base64-encoded, stored securely) Vault / credentials file
Volume Storage attached to a pod (ephemeral or persistent) A USB drive or network disk
Ingress External HTTP(S) routing rules to services The front door / reverse proxy
Node A worker machine in the cluster A server in the fleet

The pod lifecycle

Pending → Running → (CrashLoopBackOff) → Succeeded/Failed

A Deployment watches the cluster and recreates pods that fail — that is its job. You almost never create bare pods in production.

Common Operations

kubectl essentials

kubectl get pods                    # list pods in current namespace
kubectl get pods -n ml              # list in namespace "ml"
kubectl get all                     # pods, services, deployments, etc.
kubectl describe pod <name>         # detailed status + recent events
kubectl logs <pod>                  # container logs
kubectl logs <pod> -f               # follow logs (stream)
kubectl logs <pod> -c <container>   # logs of a specific container
kubectl exec -it <pod> -- bash      # shell inside the container
kubectl exec <pod> -- nvidia-smi    # run one command, no shell
kubectl apply -f file.yaml          # create/update from YAML (declarative)
kubectl delete -f file.yaml         # delete what the YAML describes
kubectl delete pod <name>           # delete one pod (deployment respawns it)
kubectl port-forward svc/api 8080:80 # tunnel local port to a service

Working with deployments

kubectl scale deploy <name> --replicas=3     # scale up/down
kubectl rollout status deploy/<name>         # watch a rollout finish
kubectl rollout history deploy/<name>        # rollout history
kubectl rollout undo deploy/<name>           # roll back to previous revision
kubectl rollout restart deploy/<name>        # restart pods (pick up new config)
kubectl set image deploy/<name> app=repo/img:v2   # update image (triggers rollout)

Logs, exec, and debugging

kubectl logs deploy/api --tail=100
kubectl logs -l app=worker --since=1h        # by label selector
kubectl exec -it deploy/api -- /bin/sh
kubectl cp <pod>:/path/file ./local          # copy files out
kubectl get events --sort-by=.lastTimestamp  # cluster events

Practical Examples

Example 1 — Deploy a model API (minimal)

# api.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-api
  namespace: ml
spec:
  replicas: 2
  selector:
    matchLabels:
      app: model-api
  template:
    metadata:
      labels:
        app: model-api
    spec:
      containers:
        - name: api
          image: your-registry/model-api:1.0
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef: { name: app-config }
            - secretRef: { name: api-keys }
          resources:
            requests: { cpu: "500m", memory: "512Mi" }
            limits: { cpu: "2", memory: "2Gi" }
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
---
apiVersion: v1
kind: Service
metadata:
  name: model-api
  namespace: ml
spec:
  selector:
    app: model-api
  ports:
    - port: 80
      targetPort: 8000

Apply and expose:

kubectl create namespace ml
kubectl apply -f api.yaml
kubectl port-forward svc/model-api 8080:80 -n ml
curl localhost:8080/health

Example 2 — GPU inference deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-server
  namespace: ml
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-server
  template:
    metadata:
      labels:
        app: llm-server
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "meta-llama/Llama-3.1-8B-Instruct", "--max-model-len", "8192"]
          resources:
            limits:
              nvidia.com/gpu: "1"      # requests a GPU via the device plugin
          volumeMounts:
            - { name: cache, mountPath: /root/.cache/huggingface }
      volumes:
        - name: cache
          persistentVolumeClaim:
            claimName: hf-cache-pvc

Key point: GPU is requested through nvidia.com/gpu as a limit (and typically a request), which requires the NVIDIA device plugin / GPU operator installed on the cluster.

Example 3 — One-off batch job

apiVersion: batch/v1
kind: Job
metadata:
  name: train-run-042
  namespace: ml
spec:
  backoffLimit: 2
  template:
    spec:
      containers:
        - name: trainer
          image: your-registry/trainer:latest
          env:
            - { name: RUN_ID, value: "042" }
      restartPolicy: Never

kubectl apply -f job.yaml
kubectl get jobs
kubectl logs job/train-run-042

Common Patterns

Config and secrets

# configmap.yaml — non-secret config
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: ml
data:
  MODEL_NAME: "bert-base-uncased"
  LOG_LEVEL: "INFO"

# secret.yaml — sensitive values (base64)
apiVersion: v1
kind: Secret
metadata:
  name: api-keys
  namespace: ml
type: Opaque
data:
  HF_TOKEN: aGZfdG9rZW5fcGxhY2Vob2xkZXI=   # echo -n value | base64

kubectl create configmap app-config --from-literal=MODEL_NAME=bert-base-uncased
kubectl create secret generic api-keys --from-literal=HF_TOKEN=YOUR_TOKEN
kubectl get secrets
kubectl describe secret api-keys

Prefer kubectl create ... --from-literal/--from-file over hand-writing base64 in YAML — it avoids accidentally committing secrets.

Namespaces for team isolation

kubectl create ns ml
kubectl -n ml get pods
kubectl config set-context --current --namespace=ml   # default namespace for the session

Scaling patterns

Pattern When Command
Manual predictable load kubectl scale deploy api --replicas=5
HPA (Horizontal Pod Autoscaler) load varies by CPU/metrics kubectl autoscale deploy api --cpu-percent=60 --min=2 --max=10
KEDA queue-based scaling (Kafka, etc.) install KEDA + ScaledObject CR

Best Practices

  • Declarative over imperative — kubectl apply -f from Git, never kubectl run/edit in prod. Store manifests in a repo and let CI apply them.
  • Pin image tags — use immutable tags (commit SHA or digest), not latest.
  • Set resource requests and limits — without them the scheduler can overcommit a node and OOM-kill neighbors.
  • Use probes — readinessProbe gates traffic during model warm-up; livenessProbe restarts hung pods. ML models with slow startup especially need readiness probes.
  • Prefer Deployments over bare pods — a Deployment gives you self-healing, rollouts, and rollback for free.
  • Don't put secrets in ConfigMaps or images — ConfigMaps are not encrypted; use Secrets (and ideally an external secrets operator / Vault).
  • Label everything — labels are how Services, HPAs, and kubectl -l selectors find resources.
  • Namespace per team/environment — dev/staging/prod separation prevents accidental cross-talk.

Common Mistakes

Mistake Consequence Correct approach
image: latest Non-reproducible, can't roll back cleanly Pin to digest or commit SHA
No resource limits Pod can consume the node, OOM-killing siblings Set requests + limits on every container
No readiness probe on model server Traffic hits pods before the model loads → timeouts Add httpGet: /health probe with generous initialDelaySeconds
Hand-writing base64 secrets in YAML Secrets committed to Git in plaintext (base64 is not encryption) kubectl create secret / external secrets
Scaling a Deployment that needs a GPU without node labels Pods stuck Pending forever Label GPU nodes (gpu=true) and use nodeSelector/nodeAffinity
Editing prod with kubectl edit Drift between Git and cluster Apply from versioned manifests only
Reading logs with kubectl logs after the pod died Logs gone with the pod Use a logging stack (Loki/CloudWatch) or terminated containers' logs

Troubleshooting

Symptom Cause Fix
Pod stuck Pending No node with enough resources / GPU kubectl describe pod — check events; add node capacity or remove limits
CrashLoopBackOff Container starts then exits kubectl logs <pod>; fix crash; kubectl get events
ImagePullBackOff Bad image name / registry auth Check the tag and pull secret (kubectl describe pod)
Pod Running but no response Model still loading / wrong port kubectl port-forward, curl health endpoint, check readiness probe path
Connection refused to a service Service selector doesn't match pod labels kubectl get pods --show-labels, verify selector matches
kubectl talks to the wrong cluster Wrong context selected kubectl config current-context, use-context the right one
GPU requested but pod Pending No nvidia.com/gpu resource registered Install NVIDIA device plugin / GPU operator on the cluster
Rollout stuck New pods unhealthy kubectl rollout status, rollout undo to revert

Official Documentation

Last Updated

2026-09-07 — Initial version.

Found a mistake or something outdated? Open an issue or submit a pull request — see CONTRIBUTING.md.