Networking in AKS

Kubenet assigns pods IPs from a private bridge network (10.244.x.x) and uses NAT for external communication, consuming only 1 VNet IP per node. Azure CNI assigns each pod a real VNet IP, enabling direct pod-to-VNet-resource communication but consuming significantly more IPs. Kubenet is simpler and cheaper; Azure CNI is needed for VNet integration, Windows nodes, and features like Virtual Nodes.

An Ingress is a Kubernetes API object that manages external HTTP/HTTPS access to services inside the cluster. It defines rules for routing traffic based on hostnames and paths. An Ingress resource needs an Ingress Controller (like NGINX or AGIC) to function — the resource alone does nothing without a controller watching it.

When you create a Service of type LoadBalancer in AKS, the Azure cloud controller manager provisions an Azure Standard Load Balancer with a public (or internal) IP. Traffic arriving at that IP on the specified port is forwarded to healthy pods matching the service's selector across the cluster nodes via node ports.

CoreDNS is the cluster DNS server in AKS. It resolves in-cluster service names like my-svc.my-namespace.svc.cluster.local to ClusterIP addresses. It also forwards external DNS queries (e.g., microsoft.com) to upstream DNS servers. Every pod's /etc/resolv.conf points to the CoreDNS service IP (typically 10.1.0.10).

A Network Policy is a Kubernetes resource that controls traffic flow at the IP/port level for pods. By default all traffic is allowed. Once a Network Policy selects a pod, only traffic explicitly allowed by the policy's rules is permitted. AKS supports Network Policies through Azure NPM, Calico, or Cilium engines.

The formula is: (max nodes × max-pods-per-node) + max nodes + 5 Azure reserved IPs. For 20 nodes with 30 pods each: 20×30 + 20 + 5 = 625 IPs, requiring at least a /22 subnet (1,022 usable). For Azure CNI Overlay, only node IPs come from the VNet subnet — pods use a separate private CIDR, drastically reducing VNet IP consumption.

A private cluster exposes the API server only on a private IP within the VNet instead of a public FQDN. Access requires being on the same network or a peered/connected network (VPN, ExpressRoute, Bastion). You manage it via a jump box VM inside the VNet or by using az aks command invoke to run kubectl commands via ARM without direct network access. CI/CD requires self-hosted agents within the VNet.

Azure NPM is Azure-managed, uses IPTables, and supports standard Kubernetes NetworkPolicy resources only. Calico supports standard NetworkPolicy plus its own GlobalNetworkPolicy CRD with richer rules (deny actions, application-layer policies, DNS-based rules). Calico is generally recommended for production workloads due to its maturity and more advanced features. NPM is being deprecated in favor of Azure Network Policy powered by Cilium.

Azure CNI Overlay assigns node IPs from the VNet subnet but pod IPs from a private, non-routable CIDR (e.g., 192.168.0.0/16). Pods can still communicate with VNet resources — traffic is NATed via the node IP. This decouples pod density from VNet IP planning, supports 250 pods per node, and drastically reduces subnet size requirements. The trade-off is that pods are not directly routable from VNet resources by their pod IP.

ExternalDNS watches Kubernetes Ingress and Service resources and automatically creates DNS records in Azure DNS zones for them. When you annotate a Service or Ingress with external-dns.alpha.kubernetes.io/hostname: myapp.example.com, ExternalDNS creates an A record in your Azure DNS zone pointing to the LoadBalancer's IP. This eliminates manual DNS management.

1) kubectl describe svc — check events for provisioning errors. 2) Verify the cluster's outbound type — if userDefinedRouting, ensure the firewall/UDR allows ARM traffic on port 443. 3) Check Azure Public IP quota in the region. 4) For internal LB, verify the subnet annotation references a real subnet with available IPs. 5) Check cloud-controller-manager logs in kube-system. 6) If the cluster uses a managed identity, verify it has Network Contributor on the VNet/subnet.

1) Check if pods are on the same node or spread across nodes — cross-node traffic adds latency. 2) Verify no Network Policies are causing dropped packets and retransmissions. 3) Check node-level CPU/memory pressure that might delay packet processing. 4) Investigate kube-proxy mode (iptables vs IPVS) — IPVS handles large service counts better. 5) Check if the service uses ExternalTrafficPolicy: Local (avoids extra hops). 6) Use kubectl exec to run curl -w with timing to isolate DNS vs connection vs transfer latency. 7) Check if CoreDNS is a bottleneck (high QPS, pods restarting).

Create a NetworkPolicy in each target namespace that allows ingress on the metrics port from the observability namespace. Example: allow ingress from namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: observability}} on the port Prometheus scrapes (usually 9090 or the application metrics port). Label the observability namespace if needed. Test with kubectl exec from a Prometheus pod to curl a target's metrics endpoint.

1) Create a private AKS cluster with --enable-private-cluster and a custom private DNS zone. 2) Deploy in a spoke VNet peered with a hub VNet. 3) The hub VNet connects to on-prem via ExpressRoute or Site-to-Site VPN. 4) Configure VNet peering between hub and spoke with "Allow forwarded traffic". 5) Link the private DNS zone to the hub VNet so the API server FQDN resolves on-prem. 6) Route cluster egress through Azure Firewall in the hub using UDR + outbound type userDefinedRouting. 7) Deploy self-hosted CI/CD agents in the spoke VNet.

1) Check CoreDNS pod health: kubectl get pods -n kube-system -l k8s-app=kube-dns — ensure all replicas are Running. 2) Check if CoreDNS pods are OOMKilled — increase memory limits. 3) Look at CoreDNS logs for upstream timeout errors — VNet custom DNS servers may be slow or unreachable. 4) Check if ndots:5 in resolve.conf causes excessive queries — add dnsConfig to pods with ndots: 2. 5) Verify the VNet DNS setting forwards to 168.63.129.16. 6) Scale CoreDNS with kubectl scale deployment coredns -n kube-system --replicas=4 if QPS is too high.