Kubernetes Plans With What You Declare
A monthly node-image upgrade took a store's checkout down for six minutes. Nothing was wrong with the upgrade. Both checkout replicas just happened to be on the same node, there was no disruption budget, and the drain evicted them together. The same cluster had pods with no resource requests, so when nodes overloaded, the autoscaler saw nothing to do.
AKS takes the control plane off your hands, and that is a big deal. What it cannot do is guess how your workloads should be scheduled, spread, protected and authenticated. Below is the cluster shape I would build today: node pools, requests and limits, disruption budgets, Workload Identity, networking choices, and what AKS Automatic sets up for you.
- You stood up an AKS cluster from the portal three years ago and haven't upgraded since
- You don't know the difference between AKS Standard and AKS Automatic
- You're still using AAD Pod Identity (deprecated) and don't know how to migrate to Workload Identity
- Your cluster has a public API server because "it was easier"
What AKS Manages, and What You Still Own
AKS gives you two SKUs in 2026:
- AKS Automatic — opinionated, production-defaults out of the box. Node Auto Provisioning (NAP) sizes nodes, security defaults enforced (Workload Identity on, local accounts off, Azure RBAC on, Defender on, image cleaner on, AKS-managed Prometheus + Grafana). Fewer knobs, faster onboarding.
- AKS Standard — full control. You pick node pools, networking, dataplane, autoscaler, RBAC mode. Required for advanced topologies (multiple node pools, GPU pools, custom CNI).
flowchart TB
subgraph Plane["Control Plane (managed by Azure)"]
API["API Server<br/>(Private Endpoint)"]
ETCD["etcd"]
SCHED["Scheduler"]
end
subgraph Nodes["Node Pools (your VNet)"]
SYS["System pool<br/>(CoreDNS, kube-proxy)"]
APP["User pool<br/>(your workloads)"]
GPU["GPU pool (taint:gpu)<br/>(autoscale 0→N)"]
end
DEV["kubectl / GitOps"] -->|Entra ID| API
APP -->|Workload Identity<br/>federated token| KV["Key Vault / Storage"]
APP -->|Azure CNI Overlay<br/>+ Cilium| EGRESS["NAT Gateway → Internet"]
APP --> ACR["Azure Container Registry<br/>(MI pull, no key)"]
style API fill:#0078D4,color:#fff,stroke:#005a9e
style KV fill:#16a34a,color:#fff,stroke:#15803d
Building a Production-Ready Cluster
Step 1: Create AKS Automatic with az CLI
# az --version >= 2.65, aks-preview ext if Automatic still gated in your region
RG=rg-aks-prod
LOC=eastus2
az group create -n $RG -l $LOC
az aks create \
-g $RG -n aks-prod \
--sku automatic \
--enable-azure-rbac \
--enable-private-cluster \
--network-plugin azure --network-plugin-mode overlay \
--network-dataplane cilium \
--enable-managed-identity \
--enable-workload-identity --enable-oidc-issuer \
--enable-azure-monitor-metrics \
--tier standard # uptime SLA-backed control planeStep 2: Configure kubectl with Entra
az aks get-credentials -g $RG -n aks-prod --overwrite-existing
# Local accounts are disabled — first kubectl call triggers an Entra device-code flow
kubectl get nodesStep 3: Workload Identity (the modern way to access Key Vault, Storage, etc.)
# 1) User-assigned managed identity for the workload
az identity create -g $RG -n mi-payments
PRINCIPAL_ID=$(az identity show -g $RG -n mi-payments --query principalId -o tsv)
CLIENT_ID=$(az identity show -g $RG -n mi-payments --query clientId -o tsv)
# 2) Federate it to a K8s service account
OIDC=$(az aks show -g $RG -n aks-prod --query oidcIssuerProfile.issuerUrl -o tsv)
az identity federated-credential create \
-g $RG --identity-name mi-payments \
--name fed-payments \
--issuer "$OIDC" \
--subject "system:serviceaccount:payments:payments-sa" \
--audience api://AzureADTokenExchange
# 3) Grant Azure RBAC on the resource (no secrets!)
az role assignment create --assignee $PRINCIPAL_ID \
--role "Key Vault Secrets User" \
--scope $(az keyvault show -g $RG -n kv-payments --query id -o tsv)# k8s/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: payments-sa
namespace: payments
annotations:
azure.workload.identity/client-id: "<CLIENT_ID-from-step-1>"
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: payments, namespace: payments }
spec:
selector: { matchLabels: { app: payments } }
template:
metadata:
labels: { app: payments, azure.workload.identity/use: "true" }
spec:
serviceAccountName: payments-sa
containers:
- name: app
image: myregistry.azurecr.io/payments:1.4.0
env:
- { name: KEY_VAULT_NAME, value: kv-payments }
# DefaultAzureCredential picks up the federated token automatically
Step 4: Node Auto Provisioning — let AKS pick the SKU
# NAP (Karpenter-based) ships in Automatic. For Standard, enable it explicitly:
# az aks update -g $RG -n aks-std --node-provisioning-mode Auto
apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: gpu-pool }
spec:
template:
spec:
requirements:
- { key: kubernetes.io/arch, operator: In, values: [amd64] }
- { key: karpenter.azure.com/sku-family, operator: In, values: [NC] }
- { key: karpenter.azure.com/sku-gpu-count, operator: Gt, values: ["0"] }
taints:
- { key: nvidia.com/gpu, value: "true", effect: NoSchedule }
limits: { cpu: "100", memory: "400Gi" }
disruption: { consolidationPolicy: WhenEmptyOrUnderutilized }Step 5: Pull from ACR with managed identity
# Attach an ACR (gives the kubelet identity AcrPull)
az aks update -g $RG -n aks-prod --attach-acr myregistry
# Push your image
az acr login -n myregistry
docker push myregistry.azurecr.io/payments:1.4.0Step 6: Upgrades — auto-upgrade channel + maintenance window
az aks update -g $RG -n aks-prod \
--auto-upgrade-channel stable \
--node-os-upgrade-channel NodeImage
az aks maintenanceconfiguration add -g $RG --cluster-name aks-prod \
--name aksManagedAutoUpgradeSchedule \
--schedule-type Weekly --day-of-week Sunday \
--interval-weeks 1 --duration 4 --start-time 02:00 --utc-offset +00:00How AKS Clusters Get Into Trouble
1. Public API server in production. A public endpoint, even with Entra auth, broadens your attack surface and complicates network policy debugging. Use --enable-private-cluster and reach it via VPN, jumpbox, or AKS Run-Command.
2. Still using AAD Pod Identity.It's retired. Workload Identity (federated tokens via OIDC) is the only supported pattern. The migration is mechanical — you create a UAMI, federate it to a service account, and the SDK picks it up via DefaultAzureCredential.
3. kubenet networking.Deprecated. Azure CNI Overlay (with Cilium dataplane) gives you per-pod IPs from a virtual address space, scales to thousands of nodes, supports Network Policy, and doesn't exhaust your VNet IPs. New clusters: always overlay.
4. Stale clusters that skip multiple minor versions.AKS supports N-2 only. If you're on 1.28 when 1.31 ships, you can't upgrade in one hop. Subscribe to the stable auto-upgrade channel with a maintenance window — the cluster keeps itself current.
5. No node pool separation. Run system pods (CoreDNS, kube-proxy, ingress controller) on a small dedicated system pool with CriticalAddonsOnly taint. User workloads on a separate user pool. Avoids resource starvation taking down DNS.
6. No Pod Disruption Budgets. Without PDBs, a node upgrade or NAP consolidation can evict every replica simultaneously and cause an outage. Define PodDisruptionBudget with minAvailable for every Deployment that matters.
What I Would Set Up First
- Default to AKS Automatic. Drop to Standard only when you need a topology Automatic doesn't support.
- Always: private API server, Azure RBAC, Workload Identity, Azure CNI Overlay + Cilium, managed Prometheus.
- Federate Workload Identity to a UAMI per workload — never store secrets in the cluster.
- Attach ACR to AKS so kubelet pulls with its identity. No image-pull secrets.
- Subscribe to the stable auto-upgrade channel and define a weekly maintenance window. Don't skip versions.
- Use NAP for elastic node provisioning instead of pinning huge static pools.
- Define PDBs for every prod workload. Run system pods on a dedicated system pool.

