Kubernetes Learnings & Reference Guide

1. Core Concepts & Hierarchy

The Structural Hierarchy

Cluster: The top-level container that holds all resources. It is the entire system (control plane + worker nodes) and the boundary of compute, networking, and storage. WHERE everything runs.
Node: A physical or virtual machine that is part of the cluster. Nodes run Pods and provide CPU, memory, and storage. WHAT runs workloads. (Check via kubectl get nodes).
Namespace: A logical partition inside a single physical cluster. Used to organize resources, separate teams, environments, or applications. Object names must be unique only within a namespace. HOW resources are grouped.
Context: A kubectl configuration that defines which cluster to talk to, which user credentials to use, and which namespace is the default. Context exists only on the client, not in the cluster. HOW kubectl connects.
Tenant (Conceptual): A team or workload owner using the cluster. Usually mapped using namespaces, RBAC, and quotas.

Labels, Names & Selectors

In Kubernetes, there are two main ways to wire resources together:

Name → Who are you? (Use a direct reference if wiring to a name).
Labels → What group do you belong to? (Use a selector to wire to labels).

Deployment Wiring Rule: The matchLabels in your selector must match the labels in your template. If they don't, the Deployment will try to create a Pod, won't recognize the Pod it just created, and will get stuck in a loop trying to create more.

template.metadata.labels: Tags every Pod generated by the manifest (e.g., app: webapp).
selector.matchLabels: Tells the controller to look for and manage any Pods with that label.

2. Architecture & Components

Control Plane

kube-api-server: Entry point to the cluster; handles all API requests and communication.
etcd: Distributed key-value store holding the entire cluster state and configuration. Highly consistent and fault-tolerant (uses Raft consensus, meaning writes require majority voting). The "brain" of the cluster.
kube-scheduler: Selects the best node for newly created Pods.
kube-controller-manager: Runs controllers that maintain the desired cluster state.
cloud-controller-manager: Connects Kubernetes with cloud provider services.

Worker Nodes

kubelet: Node agent that ensures Pods and containers run as defined.
kube-proxy: Handles networking and Service traffic routing.
CRI (Container Runtime Interface): Interface between kubelet and container runtime.
Pod: Smallest deployable unit; one or more containers sharing network and storage.

External & Other Components

Cloud Provider API: External interface for managing cloud resources (load balancers, volumes).
Admission Controller: Code in the API server that checks, modifies, or blocks requests (after auth, before etcd). Mutating (changes objects, injects sidecars) and Validating (accepts/rejects based on policy).
Provisioning: Automatically creating resources Kubernetes needs (Compute, Storage, Networking). Kubernetes asks external systems to do this; it doesn't create raw infrastructure itself.

3. Networking & Services

Basic Networking Flow

Container → Container: Same Pod (e.g., sidecar) communicate via localhost + port.
Pod → Pod: Communicate directly using pod IPs (same cluster network, no NAT). Flannel (or any CNI) is necessary for a flat IP space across different nodes for scalability.
Pod → Service: Access services via DNS name or ClusterIP/NodePort.
External → Service: Traffic reaches services using LoadBalancer or NodePort.

Service (Layer 4)

A Kubernetes Service is an internal or external network abstraction that exposes a set of Pods. You need it when Pods are replicated, may restart, or when you need stable discovery and load balancing.

A Service is a distributed, cluster-wide routing rule. It causes the cluster to write routing rules (tables) on nodes. Ingress sends traffic to the Service IP, which uses kernel routing rules to reach the Pod IP. The Service IP is a virtual IP that exists conceptually in the cluster.

Port Definitions

Port Type	Analogy	Description
`targetPort`	Internal room number in a building	The Pod container port (e.g., nginx port)
`port`	Building's internal doorway	The K8s Service port (inside the cluster)
`nodePort`	Main gate from the street	External access port opened on every node (Range 30000–32767)

Service Types Explained

ClusterIP: Accessible only from inside the Kubernetes cluster.
NodePort: Opens a port on all nodes. kube-proxy forwards traffic from the NodeIP:NodePort to backend Pods. (Pods only listen on internal IP/port and don't know about NodePort).
LoadBalancer: Creates a cloud provider load balancer (AWS ELB, GCP LB, Azure LB).

Ingress (Layer 7)

Alternative to LoadBalancer and NodePort to expose services outside the cluster. It provides HTTP/HTTPS routing, TLS termination, and virtual hosts. Requires an Ingress Controller (e.g., NGINX, Traefik) which runs as pods inside the cluster.

LoadBalancer vs Ingress

Every service exposed externally gets its own cloud load balancer. Typically, you deploy one external load balancer in front of the Ingress Controller to provide a stable internet entry point. The Ingress Controller then routes traffic internally to many services.

The Routing Chain

IngressController → watches → Ingress → points to → Service → selects → Pods
Deployment → manages → Pods

Data Flow: Ingress → Service → Secret (for TLS)
Resource Usage: Pods/Deployments/Ingress → may use → Secrets

Service Discovery & DNS

Allows apps to find each other automatically. Kubernetes injects a custom /etc/resolv.conf into every Pod. This points to the internal DNS service (CoreDNS) and includes search domains (e.g., default.svc.cluster.local), allowing you to curl by service name.

Cross-Namespace Rules:

Service selectors never cross namespaces. You cannot have a Service in Namespace A select Pods in Namespace B.
However, DNS lets you reach Services anywhere. Use the DNS name: <service>.<namespace>.svc.cluster.local
DNS resolves to Services; Services select Pods locally. Pods never get DNS records.

ExternalDNS: Automates DNS updates for external providers (AWS Route 53, Cloudflare) when services/ingresses get new IPs.

4. Workload Management

Workload Types

Deployment: Defines a template for Pods. It creates Pods; Services select them. Deployments don't run containers themselves, they manage the Pods that do.
StatefulSets: Manage stateful apps requiring stable network identities, persistent storage, and ordered deployment/scaling (e.g., databases).
DaemonSets: Ensures a specific pod runs on every (or selected) node (e.g., log collectors, monitoring).
Autoscaling: HPA (Horizontal Pod Autoscaler) or VPA (Vertical) adjusts based on CPU/memory.

Probes

Liveness Probe: "Is this container still alive?"
Readiness Probe: "Is this container ready to receive traffic?"

Volumes in Deployments

A volume defined in a Deployment (Pod spec) is available to all containers, but is mounted only in the containers that list it in volumeMounts.

5. Scheduling & Placement

nodeSelector: A strict rule (exact match) mapping Pods to nodes using key-value labels (e.g., hardware=high-spec).
Affinity / Anti-Affinity: Controls which nodes a pod prefers or requires (or avoids) running on.
Inter-Pod Affinity / Anti-Affinity: Schedules pods near (or away from) other specific pods for performance or high availability.
Taints and Tolerations: Taints prevent pods from being scheduled on specific nodes. Tolerations allow specific pods to bypass those taints.

6. Configuration, Storage & Security

Configuration & Storage

ConfigMap: Stores config data (env vars, config files) separate from app code.
Volumes Auto-Provisioning: Creates storage dynamically using StorageClasses when a pod requests it.
PodPresets: Automatically injects env vars, volumes, or secrets at creation time (Deprecated in newer K8s versions).

Secrets

Can be created via YAML (using Base64 encoding) or CLI:

# CLI Example for DB Passwords:
echo -n "root" > ./username.txt
echo -n "password" > ./password.txt
kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt

# CLI Example for SSL:
kubectl create secret generic ssl-certificate --from-file=ssh-privatekey=~/.ssh/id_rsa --ssl-cert-=ssl-cert=mysslcert.crt

Security & Resource Limits

ResourceQuota: Controls CPU/memory per namespace. (Requests = guaranteed; Limits = maximum).
User Management: Authentication via Bearer tokens, OIDC, or internal users (Secrets).
RBAC: Role-Based Access Control defines who can do what.
Pod Security Policy / Admission: Enforces rules on how pods run (e.g., "must not run as root").

7. Extensions

CRDs (Custom Resource Definitions): Allow users to create custom resource types, extending the K8s API.
Operators: Controllers that automate complex app management using CRDs and custom logic (deployment, backup, scaling).

8. Common Troubleshooting & Commands

kubectl get pod -o wide: Lists Pods with additional details (Node, IP, etc.).
SCALING-ERROR: A Service is not scalable. If you put a Deployment and a Service in the same file and try to scale it, it will fail (Error: NotFound). Fix: Put Deployment and Service in separate YAML files.
NGINX Port Issue: Running kubectl expose deployment hello-minikube --type=NodePort --port=8080 without a --target-port assumes the container listens on 8080. If your image listens on 80, connections will reset.