# KUBERNETES POD MASTERY — CKA REVISION NOTES
GOAL:
Master Pod YAML, container configuration, volumes, Downward API, probes, resources, security, lifecycle, networking, and troubleshooting.
==================================================
1. BASIC POD STRUCTURE
==================================================
apiVersion: v1
kind: Pod
metadata:
name: mypod
namespace: pod-mastery
labels:
app: demo
spec:
restartPolicy: Always
containers:
- name: nginx
image: nginx:1.27
ports:
- containerPort: 80
IMPORTANT FIELDS:
apiVersion: API version
kind: Resource type
metadata: Name, namespace, labels, annotations
spec: Desired configuration
status: Actual observed state
FIELDS UNDER SPEC:
containers, initContainers, volumes, restartPolicy,
securityContext, nodeSelector, affinity, tolerations,
dnsPolicy, serviceAccountName, terminationGracePeriodSeconds.
FIELDS UNDER CONTAINER:
image, command, args, env, envFrom, ports, resources,
volumeMounts, probes, lifecycle, securityContext,
workingDir, imagePullPolicy.
COMMANDS:
kubectl apply -f pod.yaml
kubectl get pods -n pod-mastery -o wide
kubectl describe pod mypod -n pod-mastery
kubectl get pod mypod -n pod-mastery -o yaml
kubectl delete pod mypod -n pod-mastery
FIELD DISCOVERY:
kubectl explain pod.spec
kubectl explain pod.spec.containers
kubectl explain pod.spec.volumes
kubectl explain pod.spec.containers.env
kubectl explain pod.spec.containers.resources
==================================================
2. LABELS, SELECTORS AND ANNOTATIONS
==================================================
metadata:
labels:
app: payment
tier: backend
annotations:
description: Payment application
kubectl get pods --show-labels
kubectl get pods -l app=payment
kubectl label pod mypod environment=dev
kubectl annotate pod mypod owner=platform-team
Labels identify and select resources.
Annotations store additional metadata.
==================================================
3. COMMAND AND ARGS
==================================================
containers:
- name: worker
image: busybox:1.36
command: ["/bin/sh", "-c"]
args:
- 'echo "Starting"; sleep 3600'
command overrides the image entrypoint.
args overrides the default command arguments.
Shell syntax requires a shell such as /bin/sh -c.
==================================================
4. ENVIRONMENT VARIABLES
==================================================
Direct environment variable:
env:
- name: APP_ENV
value: production
ConfigMap envFrom:
envFrom:
- configMapRef:
name: app-config
One ConfigMap key:
env:
- name: DB_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: database-host
Secret key:
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-secret
key: password
Create resources:
kubectl create configmap app-config --from-literal=APP_ENV=production
kubectl create secret generic db-secret --from-literal=password=example-password
Environment variables do not automatically update when the referenced ConfigMap or Secret changes.
==================================================
5. DOWNWARD API: FIELDREF AND RESOURCEFIELDREF
==================================================
fieldRef exposes Pod metadata or fields.
env:
- name: MY_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MY_NODE
valueFrom:
fieldRef:
fieldPath: spec.nodeName
Other useful fields:
metadata.name
metadata.labels
metadata.annotations
spec.serviceAccountName
resourceFieldRef exposes a container resource request or limit.
containers:
- name: app
image: busybox:1.36
command: ["/bin/sh", "-c", "sleep 3600"]
resources:
limits:
memory: 200Mi
env:
- name: MEMORY_LIMIT
valueFrom:
resourceFieldRef:
containerName: app
resource: limits.memory
Downward API volume:
volumes:
- name: pod-info
downwardAPI:
items:
- path: pod-name
fieldRef:
fieldPath: metadata.name
- path: pod-namespace
fieldRef:
fieldPath: metadata.namespace
Mount it:
volumeMounts:
- name: pod-info
mountPath: /etc/pod-info
readOnly: true
Read:
kubectl exec mypod -- cat /etc/pod-info/pod-name
Remember:
fieldRef = Pod fields.
resourceFieldRef = container resource requests or limits.
==================================================
6. IMAGE SETTINGS, PORTS AND WORKING DIRECTORY
==================================================
containers:
- name: app
image: nginx:1.27
imagePullPolicy: IfNotPresent
workingDir: /tmp
ports:
- name: http
containerPort: 8080
protocol: TCP
Image pull policies:
Always: resolve/pull image when starting.
IfNotPresent: use a cached image when available.
Never: do not pull an image.
containerPort documents the container port; it does not expose the Pod externally.
Private registry example:
spec:
imagePullSecrets:
- name: registry-credentials
containers:
- name: app
image: registry.example.com/team/app:v1
The imagePullSecret must exist in the Pod namespace.
==================================================
7. MULTI-CONTAINER PODS, INIT CONTAINERS AND SIDECARS
==================================================
Containers in a Pod share the network namespace and Pod IP.
They can communicate through localhost.
They do not automatically share their filesystems.
Init container example:
spec:
initContainers:
- name: init
image: busybox:1.36
command: ["/bin/sh", "-c", "echo initialized > /shared/init.txt"]
volumeMounts:
- name: shared-data
mountPath: /shared
containers:
- name: app
image: busybox:1.36
command: ["/bin/sh", "-c", "cat /shared/init.txt; sleep 3600"]
volumeMounts:
- name: shared-data
mountPath: /shared
volumes:
- name: shared-data
emptyDir: {}
An init container must complete successfully before ordinary app containers start.
Native sidecar example:
initContainers:
- name: helper
image: busybox:1.36
restartPolicy: Always
command: ["/bin/sh", "-c", "while true; do sleep 10; done"]
Native sidecars are restartable init containers. Confirm cluster version and feature support.
==================================================
8. POD VOLUMES
==================================================
Volume types:
emptyDir: Temporary Pod-scoped storage.
hostPath: Mount a node filesystem path.
configMap: Configuration files.
secret: Sensitive configuration files.
persistentVolumeClaim: Persistent storage claim.
projected: Combine multiple volume sources.
downwardAPI: Expose Pod metadata as files.
A. EMPTYDIR
volumes:
- name: shared
emptyDir: {}
Mount into each container:
volumeMounts:
- name: shared
mountPath: /shared
Test shared storage:
kubectl exec mypod -c writer -- sh -c 'echo hello > /shared/test.txt'
kubectl exec mypod -c reader -- cat /shared/test.txt
emptyDir survives a container restart within the same Pod but is lost when the Pod is removed.
B. HOSTPATH
volumes:
- name: node-data
hostPath:
path: /data/app
type: DirectoryOrCreate
hostPath depends on the node and has security risks.
C. CONFIGMAP AND SECRET VOLUMES
volumes:
- name: config-files
configMap:
name: app-config
- name: secret-files
secret:
secretName: db-secret
Mount:
volumeMounts:
- name: config-files
mountPath: /etc/app-config
readOnly: true
- name: secret-files
mountPath: /etc/app-secret
readOnly: true
Keys generally become files in the mount directory.
D. SUBPATH
volumeMounts:
- name: config-files
mountPath: /etc/app/settings.conf
subPath: settings.conf
subPath mounts a specific file or directory.
subPath mounts do not receive normal projected ConfigMap/Secret updates.
E. PROJECTED VOLUME
volumes:
- name: combined-config
projected:
sources:
- configMap:
name: app-config
- secret:
name: db-secret
- downwardAPI:
items:
- path: pod-name
fieldRef:
fieldPath: metadata.name
Projected volumes combine multiple sources into one directory.
F. PERSISTENT VOLUME AND PVC
PVC example:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-pvc
namespace: pod-mastery
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
This requires a suitable StorageClass/default provisioner or a matching pre-created PV.
Mount the PVC:
spec:
containers:
- name: app
image: busybox:1.36
command: ["/bin/sh", "-c", "echo persistent > /data/test.txt; sleep 3600"]
volumeMounts:
- name: app-storage
mountPath: /data
volumes:
- name: app-storage
persistentVolumeClaim:
claimName: app-pvc
Check:
kubectl get pv
kubectl get pvc -n pod-mastery
kubectl describe pvc app-pvc -n pod-mastery
emptyDir belongs to the Pod lifetime.
PVC storage can persist beyond an individual Pod, depending on the storage backend and reclaim configuration.
==================================================
9. PROBES
==================================================
Startup probe: Determines whether startup has completed.
Readiness probe: Determines whether the Pod should receive traffic.
Liveness probe: Detects a container that should be restarted.
Example:
containers:
- name: app
image: nginx:1.27
ports:
- containerPort: 80
startupProbe:
httpGet:
path: /
port: 80
periodSeconds: 5
failureThreshold: 30
readinessProbe:
httpGet:
path: /
port: 80
periodSeconds: 5
livenessProbe:
httpGet:
path: /
port: 80
periodSeconds: 10
Probe handlers:
httpGet
tcpSocket
exec
grpc, when supported
Exec probe example:
readinessProbe:
exec:
command: ["/bin/sh", "-c", "test -f /tmp/ready"]
periodSeconds: 5
Readiness failure marks the Pod not ready.
Liveness failure can restart the container.
Startup failure can restart the container after the threshold is reached.
Startup success enables the other probes.
==================================================
10. RESOURCES AND QOS
==================================================
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 200m
memory: 200Mi
100m CPU = 0.1 CPU core.
CPU limits can cause throttling.
Memory limit exceedance can cause OOMKilled.
QoS classes:
Guaranteed: Every container has CPU and memory requests equal to limits.
Burstable: Some resource requests/limits exist but Guaranteed criteria are not met.
BestEffort: No CPU or memory requests or limits for any container.
Check:
kubectl describe pod mypod
kubectl get pod mypod -o jsonpath='{.status.qosClass}{"
"}'
Ephemeral storage:
resources:
requests:
ephemeral-storage: 100Mi
limits:
ephemeral-storage: 500Mi
Node disk pressure can lead to Pod eviction.
==================================================
11. SECURITY CONTEXT
==================================================
Pod-level security:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 2000
Container-level security:
containers:
- name: app
image: nginxinc/nginx-unprivileged
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
Important:
runAsUser: Process UID.
runAsGroup: Primary process GID.
runAsNonRoot: Require non-root execution.
fsGroup: Group-related access for supported volumes.
allowPrivilegeEscalation: Controls privilege escalation.
readOnlyRootFilesystem: Makes the container root filesystem read-only.
capabilities: Controls Linux capabilities.
seccompProfile: Restricts system calls.
Read-only filesystems may require writable temporary volumes.
Volume permissions must be compatible with the chosen UID/GID.
POD SECURITY STANDARDS:
Privileged: Few restrictions.
Baseline: Prevents known privilege-escalation patterns.
Restricted: Stronger security requirements.
POD SECURITY ADMISSION:
enforce: Reject violating Pods.
audit: Record violations.
warn: Warn about violations.
Example:
kubectl label namespace pod-mastery pod-security.kubernetes.io/warn=restricted
==================================================
12. LIFECYCLE AND GRACEFUL TERMINATION
==================================================
containers:
- name: app
image: nginx:1.27
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo started"]
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30
postStart runs after container creation but is not guaranteed to finish before the entrypoint starts.
preStop runs during termination and consumes the grace period.
Processes should handle termination signals properly.
activeDeadlineSeconds limits the total active lifetime of a Pod.
==================================================
13. RESTART POLICIES AND TROUBLESHOOTING
==================================================
restartPolicy values:
Always: Restart exited containers; default.
OnFailure: Restart containers that exit unsuccessfully.
Never: Do not restart exited containers.
Common symptoms:
Pending:
Check scheduling events, resource requests and PVC binding.
ErrImagePull/ImagePullBackOff:
Check image name, registry connectivity and credentials.
CrashLoopBackOff:
Check logs, exit code, command and probes.
CreateContainerConfigError:
Check ConfigMaps, Secrets and referenced keys.
OOMKilled:
Check memory limits and application memory usage.
Running but not Ready:
Check readiness probe and application listener.
Evicted:
Check node memory, disk and ephemeral-storage pressure.
Commands:
kubectl get pods -o wide
kubectl describe pod mypod
kubectl logs mypod -c app
kubectl logs mypod -c app --previous
kubectl exec mypod -c app -- sh
kubectl get events --sort-by=.metadata.creationTimestamp
For multi-container Pods, use -c to select the container.
--previous retrieves logs from the previous container instance.
==================================================
14. DNS, HOSTNAME AND NETWORKING
==================================================
spec:
hostname: app-host
subdomain: app-headless
dnsPolicy: ClusterFirst
DNS policies:
ClusterFirst: Cluster DNS; normal default.
Default: Inherit node DNS settings.
ClusterFirstWithHostNet: Cluster DNS behavior with host networking.
None: Use explicit DNS configuration.
Other fields:
spec:
hostNetwork: false
hostPID: false
hostIPC: false
hostAliases:
- ip: 192.0.2.10
hostnames:
- internal.example
Containers in a Pod share a network namespace, Pod IP and port space.
hostNetwork, hostPID and hostIPC can expose the Pod to host-level resources.
==================================================
15. SERVICE ACCOUNTS
==================================================
spec:
serviceAccountName: app-service-account
automountServiceAccountToken: false
Create:
kubectl create serviceaccount app-service-account -n pod-mastery
kubectl get serviceaccount -n pod-mastery
ServiceAccount provides identity, not permissions by itself.
RBAC controls permissions.
Disable automatic token mounting if the application does not need API access.
==================================================
16. SCHEDULING-RELATED POD FIELDS
==================================================
nodeSelector:
spec:
nodeSelector:
kubernetes.io/hostname: workernode2
Toleration:
spec:
tolerations:
- key: dedicated
operator: Equal
value: app
effect: NoSchedule
Other fields to recognize:
affinity
anti-affinity
topologySpreadConstraints
priorityClassName
runtimeClassName
nodeName
A toleration permits matching taints but does not guarantee scheduling onto a node.
==================================================
17. STATIC PODS AND KUBELET
==================================================
Common kubeadm configuration:
staticPodPath: /etc/kubernetes/manifests
Commands:
systemctl status kubelet
systemctl cat kubelet
cat /var/lib/kubelet/config.yaml
ls -l /etc/kubernetes/manifests/
A Static Pod is managed directly by kubelet from a local manifest.
Many kubeadm control-plane components run as Static Pods.
The path is installation-specific.
==================================================
18. TROUBLESHOOTING WORKFLOW
==================================================
Step 1:
kubectl get pod mypod -o wide
Step 2:
kubectl describe pod mypod
Step 3:
kubectl logs mypod -c app
kubectl logs mypod -c app --previous
Step 4:
kubectl get pod mypod -o yaml
kubectl explain pod.spec.containers
Check:
Field spelling and nesting.
Image and pull credentials.
ConfigMap/Secret names and keys.
Volume definitions and mounts.
Probe configuration.
Resource requests and limits.
Security context and file permissions.
Events and container exit codes.
==================================================
19. ULTIMATEPOD PRACTICE ALREADY COMPLETED
==================================================
Your integrated Pod included:
- nginx-unprivileged main container.
- BusyBox sidecar.
- ConfigMap envFrom.
- ConfigMap volume.
- Shared emptyDir.
- HTTP readiness and liveness probes.
- Exec probes for the sidecar.
- CPU/memory requests and limits.
- Pod securityContext with UID 1000, GID 1000 and fsGroup.
- allowPrivilegeEscalation: false.
Observed result:
Pod Running, 2/2 Ready, Guaranteed QoS.
ConfigMap-mounted APP_ENV file contained production.
Lessons:
Use ls to inspect directories and cat to read files.
Container filesystems are separate unless shared volumes are mounted.
A file under sidecar /tmp is not automatically visible to the main container.
Use a shared volume when containers need to exchange files.
==================================================
20. FINAL COMMAND CHEAT SHEET
==================================================
List Pods:
kubectl get pods -o wide
Inspect Pod:
kubectl describe pod mypod
View YAML:
kubectl get pod mypod -o yaml
Container logs:
kubectl logs mypod -c app
Previous logs:
kubectl logs mypod -c app --previous
Enter container:
kubectl exec -it mypod -c app -- sh
Inspect fields:
kubectl explain pod.spec.containers
ConfigMaps and Secrets:
kubectl get configmap,secret
Storage:
kubectl get pv,pvc
Events:
kubectl get events --sort-by=.metadata.creationTimestamp
Client-side dry run:
kubectl apply --dry-run=client -f pod.yaml
Generate Pod YAML:
kubectl run testpod --image=nginx --dry-run=client -o yaml
==================================================
21. HOW TO MASTER THE 60 PRACTICAL QUESTIONS
==================================================
1. Try writing YAML from memory.
2. If stuck, consult kubectl explain or official Kubernetes documentation.
3. Use AI to understand a field or correct an error, not just copy a solution.
4. Apply the manifest and verify the actual behavior.
5. Inspect events, logs, status and generated YAML.
6. Record recurring mistakes and repeat those questions.
7. Practise combining Downward API, PVCs, projected volumes, init containers, sidecars, probes and security settings.
FINAL CKA RULE:
Understand what each field does, where it belongs, how to find it and how to verify its effect. You do not need to memorize every field perfectly before practising.
# Initialize & Clone
git init # Create a new repository in current folder
git clone <url> # Copy a remote repo locally
# Status & Commit
git status # Show repo state (staged/unstaged files)
git add . # Stage all changes
git commit -m "msg" # Save snapshot with message
git log --oneline # Compact commit history
git diff # Show changes not staged/committed
# Branching
git branch # List branches
git branch <name> # Create new branch
git checkout <name> # Switch branch
git checkout -b <name> # Create + switch branch
git merge <branch> # Merge branch into current
git rebase <branch> # Reapply commits on top of branch
# Remote
git remote -v # Show remotes
git remote add origin <url> # Add remote repo
git push origin main # Push changes to remote main
git pull origin main # Fetch + merge from remote main
git fetch origin # Fetch without merging
# Undo & Reset
git reset <file> # Unstage file
git checkout -- <file> # Discard local changes
git revert <commit> # Undo commit safely (new commit)
git reset --hard <commit> # Reset repo to commit (dangerous)
# Stash
git stash # Save changes temporarily
git stash list # Show stashes
git stash apply # Reapply latest stash
git stash drop # Delete latest stash
Linux Kernel — Quick Summary
Kernel: Core component of the OS that acts as a bridge between applications and hardware and manages CPU, memory, filesystems, devices, networking, etc.
Types: Monolithic – major OS functions in kernel space; Microkernel – only essential functions; Hybrid – combination of both; Exokernel – minimal kernel with more direct hardware control.
Kernel Modules: Pieces of kernel code, commonly drivers, that can be dynamically loaded/unloaded without rebooting.
Commands: lsmod → view modules, modprobe module → load, modprobe -r module → remove.
Linux: A monolithic kernel with support for loadable kernel modules.
=========================================================================================================================================
RPO vs RTO — Quick Summary
Term Full Form Simple Meaning Example
RPO Recovery Point Objective How much data can we afford to lose? RPO = 15 min → maximum 15 min of data loss
RTO Recovery Time Objective How quickly must the service be restored? RTO = 2 hours → service must recover within 2 hours
RPO defines the maximum acceptable data loss, while RTO defines the maximum acceptable downtime required to restore the service.
==============================================================================================================
RHEL Boot Process — Quick Summary
BIOS / UEFI
↓
MBR / EFI System Partition
↓
GRUB2
↓
Kernel + initramfs
↓
Kernel initialization
↓
Real root filesystem
↓
systemd (PID 1)
↓
Targets
↓
Services
↓
Login
BIOS/UEFI → Initializes hardware and finds the bootloader.
GRUB2 → Loads the Linux kernel + initramfs.
Kernel → Initializes CPU, memory, devices and other kernel components.
initramfs → Provides drivers/tools to find and mount the real root filesystem.
systemd (PID 1) → Starts userspace services and reaches the configured target.
Target → Brings the system to the required state (multi-user.target / graphical.target).
Finally, services start and the login prompt appears.
===============================================================================
Linux File Types — Quick Summary
- → Regular file → Normal data/config/log files
d → Directory → Contains files/directories
l → Symbolic link → Shortcut/reference to another file
c → Character device → Character/stream-based device
b → Block device → Storage devices like HDD/SSD
s → Socket → Communication between processes
p → Named pipe → Communication between processes using FIFO
================================================================================================
Cron / Crontab — Quick Summary
Cron → Time-based job scheduler used to run commands/scripts automatically.
crond → Background daemon/service that executes cron jobs.
crontab → File containing scheduled jobs.
Basic commands
crontab -l # List jobs
crontab -e # Edit jobs
crontab -r # Remove crontab
crontab -u user -l # View another user's crontab
* * * * * command
│ │ │ │ │
│ │ │ │ └─ Day of week (0-7)
│ │ │ └─── Month (1-12)
│ │ └───── Day of month (1-31)
│ └─────── Hour (0-23)
└───────── Minute (0-59)
0 2 * * * /backup.sh
* → every
, → multiple values
- → range
/ → interval
→ Every 10 minutes.
*/10 * * * * command
Important locations
/var/spool/cron/ → User crontabs
/etc/crontab → System-wide cron
/etc/cron.d/ → System cron jobs
/etc/cron.daily/ → Daily jobs
/etc/cron.weekly/ → Weekly jobs
/etc/cron.monthly/ → Monthly jobs
User crontab:
* * * * * command
/etc/crontab:
* * * * * user command
systemctl status crond
journalctl -u crond
===========================================================================================================================
RHEL UsrMerge — Quick Summary
Modern RHEL uses UsrMerge, where many traditional directories are symlinks to locations under /usr.
/bin → /usr/bin
/sbin → /usr/sbin
/lib → /usr/lib
/lib64 → /usr/lib64
l at the beginning of ls -l → symbolic link
The actual target directories are under /usr.
Old paths are kept as symlinks for backward compatibility.
Example:
/bin/ls
→ follows the symlink → /usr/bin/ls.
Remember: /usr is the main location; /bin, /sbin, /lib, and /lib64 are compatibility links on modern RHEL.
Linux Directory Structure — Quick Summary
/ → Root of entire filesystem
/boot → Kernel, initramfs, GRUB boot files
/etc → System configuration files
/home → Normal users' home directories
/root → Root user's home directory
/bin → Symlink → /usr/bin
/sbin → Symlink → /usr/sbin
/usr → Programs, libraries, shared data
/var → Logs, cache, variable/application data
/tmp → Temporary files
/dev → Device files
/proc → Process & kernel information (virtual)
/sys → Hardware/device & kernel information (virtual)
/run → Runtime data since boot
/opt → Optional/third-party applications
/mnt → Temporary filesystem mount point
/media → Removable media
/srv → Data provided by services
/etc = Configuration | /var = Logs/Data | /boot = Boot | /usr = Programs | /home = Users | /dev = Devices | /proc = Processes | /sys = Hardware | /tmp = Temporary.
=====================================================================================
Linux Links + Inode — Quick Summary
🔹 Inode
Inode is a filesystem data structure that stores file metadata and references to the file's data blocks.
Filename → Inode → File Data
Inode contains:
Permissions
Owner/group
Size
Timestamps
Link count
Data block references
Filename is mapped to the inode by the directory.
🔹 Hard Link
Another filename pointing to the same inode.
file1 ──→ inode 123
file2 ──→ inode 123
Same inode
Same data
Cannot normally cross filesystem
Normally cannot link directories
Original filename deleted → hard link still works
ln file1 file2
Inode → File metadata + data references
Hard link → SAME inode
Soft link → DIFFERENT inode → PATH
======================================================================================================================
/etc/fstab (filesystem table) contains persistent filesystem mount configuration, so filesystems can be mounted automatically during boot.
UUID/device | Mount Point | FS Type | Options | Dump | Pass
UUID=abc123 /data xfs defaults 0 0
UUID/device → Identifies the filesystem.
Mount point → Where it will be mounted, e.g. /data.
FS type → xfs, ext4, etc.
Options → How to mount, e.g. defaults, ro, rw, nofail.
Dump → Legacy backup setting; commonly 0.
Pass → Filesystem check order; commonly 0 for XFS.
fstab = What to mount + Where to mount + Filesystem type + How to mount + Boot/check settings.
============================================================================================================================
NFS — Quick Summary
NFS (Network File System) → Allows a client to access a directory/filesystem from a remote Linux server over the network.
NFS Server → Shares/exports a directory using /etc/exports.
NFS Client → Mounts the exported directory.
dnf install nfs-utils
Server → nfs-utils + nfs-server
Client → nfs-utils
dnf install nfs-utils
Server → nfs-utils + nfs-server
Client → nfs-utils
Firewall
firewall-cmd --permanent --add-service=nfs
firewall-cmd --reload
Server
/etc/exports
/data 192.168.1.20(rw,sync)
exportfs -rav
exportfs -v
mount -t nfs server:/data /mnt/data
/etc/fstab
server:/data /mnt/data nfs defaults,_netdev 0 0
_netdev → tells systemd that the filesystem requires the network.
rw → Read/write
ro → Read-only
sync → Synchronous writes
async → Asynchronous writes
root_squash → Client root mapped to unprivileged user
no_root_squash → Client root not squashed
all_squash → All users mapped to anonymous user
NFSv4 → TCP 2049
NFSv3 → 2049 + RPC services such as rpcbind (111)
Important commands
systemctl status nfs-server
exportfs -v
exportfs -rav
showmount -e server
mount
umount
findmnt
rpcinfo -p server
journalctl -u nfs-server
NFS = Server exports → Client mounts → /etc/exports controls sharing → /etc/fstab makes it persistent → nfs-utils provides NFS tools.
===============================================================================================================================
RHEL PACKAGE MANAGEMENT — INTERVIEW QUICK NOTES
“yum update updates installed packages while generally preserving obsolete packages. yum upgrade also updates packages but can remove obsolete packages when required. For normal maintenance, yum update is commonly used. yum upgrade is useful when you want the installed package set to reflect the current repository state.”
1. RPM vs YUM vs DNF
--------------------
RPM = Low-level package management tool
YUM = Older high-level package manager
DNF = Modern high-level package manager / successor to YUM
RHEL 7 → YUM commonly used
RHEL 8+ → DNF primarily used
RPM works directly with .rpm files.
DNF/YUM work with repositories and resolve dependencies.
IMPORTANT:
Don't say "yum is simply a symlink to dnf".
In modern RHEL, yum is a compatibility interface using DNF
functionality.
2. RPM IMPORTANT COMMANDS
-------------------------
Install:
rpm -ivh package.rpm
Upgrade:
rpm -Uvh package.rpm
Remove:
rpm -e package
List installed:
rpm -qa
Package information:
rpm -qi package
Files installed by package:
rpm -ql package
Find which package owns a file:
rpm -qf /path/file
Show dependencies:
rpm -qR package
Verify package:
rpm -V package
3. DNF IMPORTANT COMMANDS
-------------------------
Install:
dnf install package
Remove:
dnf remove package
Update:
dnf update
Search:
dnf search package
Package information:
dnf info package
Find package providing command/file:
dnf provides '*/command'
Reinstall:
dnf reinstall package
List repositories:
dnf repolist
Clean cache:
dnf clean all
Refresh metadata:
dnf makecache
Transaction history:
dnf history
Undo transaction:
dnf history undo <ID>
4. REPOSITORIES
-------------------------------------------------------------------------------------------
Repository = location containing RPM packages + metadata.
Repository configuration:
/etc/yum.repos.d/*.repo
DNF main configuration:
/etc/dnf/dnf.conf
Important .repo options:
baseurl = repository location
enabled=1 = repository enabled
gpgcheck=1 = verify package signature
5. DEPENDENCY
-------------------------------------------------------------------------------------------
Dependency = another package/library required by software.
RPM:
- Directly installs RPM
- Does not provide DNF-style automatic repository
dependency resolution
DNF:
- Finds package from repository
- Resolves dependencies automatically
6. GPG
-------------------------------------------------------------------------------------------
GPG verifies package signatures.
gpgcheck=1
= Enable package signature verification.
7. RPM DATABASE
-------------------------------------------------------------------------------------------
Location:
/var/lib/rpm/
Contains information about installed RPM packages.
8. RHEL REPOSITORY/SUBSCRIPTION
-------------------------------------------------------------------------------------------
Check subscription:
subscription-manager status
List repositories:
subscription-manager repos --list
Enable repository:
subscription-manager repos --enable=<repo-id>
9. COMMON INTERVIEW SCENARIOS
-------------------------------------------------------------------------------------------
Which package installed a file?
rpm -qf /path/file
Which package provides a command?
dnf provides '*/command'
What files did a package install?
rpm -ql package
Why does DNF say "No match"?
→ Check repository, subscription, metadata and package availability.
Refresh DNF metadata:
dnf clean all
dnf makecache
See recent package changes:
dnf history
⭐ MOST IMPORTANT MEMORY
RPM → Direct .rpm package management
DNF → Repository + dependency management
YUM → Older DNF-style package manager
Repo → Source of RPM packages
GPG → Package signature verification
/var/lib/rpm/ → RPM database
/etc/yum.repos.d/ → Repository files
/etc/dnf/dnf.conf → DNF configuration
RPM → rpm -qa / -qi / -ql / -qf
DNF → install / remove / update / search / provides / history
=============================================================================================
SSHD + PUBLIC KEY AUTHENTICATION
================================
SSH = client
SSHD = SSH server/daemon
Default port = TCP 22
Main config:
/etc/ssh/sshd_config
Important settings:
Port 22
PermitRootLogin no
PasswordAuthentication yes/no
PubkeyAuthentication yes
AllowUsers / AllowGroups
Service:
systemctl status sshd
systemctl restart/reload sshd
Test config:
sshd -t
PUBLIC KEY AUTHENTICATION
=========================
Key pair:
Private key → Client (KEEP SECRET)
Public key → Server
Create:
ssh-keygen -t ed25519
Copy public key:
ssh-copy-id user@server
Server stores public key in:
~/.ssh/authorized_keys
Permissions:
~/.ssh = 700
authorized_keys = 600
private key = 600
Authentication:
Client proves it has private key
↓
Server checks public key in authorized_keys
↓
Login successful
known_hosts = Client stores server host keys
authorized_keys = Server stores user public keys
Troubleshooting:
ssh -vvv user@server
journalctl -u sshd
ss -lntp | grep :22
============================================================================================================================
RSYNC — SHORT INTERVIEW NOTES
============================================================================================================================
Rsync = File/directory synchronization tool.
Used for backup, migration and repeated data transfer.
Basic:
rsync -av /data/ /backup/
Remote:
rsync -av /data/ user@server:/backup/
Important options:
-a = archive/preserve attributes
-v = verbose
-z = compression
-P = progress + partial
-n = dry run
--delete = delete destination files not in source
--exclude = exclude files
Rsync over SSH → TCP 22
Rsync daemon → TCP 873
Daemon config → /etc/rsyncd.conf
IMPORTANT:
/data → copy directory
/data/ → copy contents
RSYNC vs SCP
==============================================================================================================================
SCP = Secure file copy
Rsync = File synchronization
SCP → simple/one-time copy
Rsync → repeated sync, transfers only required changes,
supports --delete, --exclude, --dry-run, etc.
INTERVIEW:
"SCP is for secure copying; rsync is for efficient
synchronization and repeated transfers."
===============================================================================================================================
LVM = Logical Volume Manager
Purpose = Flexible storage management
Architecture:
Disk/Partition → PV → VG → LV → Filesystem → Mount Point
PV = Physical Volume
Physical disk/partition prepared for LVM
Command: pvcreate
VG = Volume Group
Storage pool made from one or more PVs
Command: vgcreate
LV = Logical Volume
Virtual disk created from VG
Command: lvcreate
PE = Physical Extent
LE = Logical Extent
PV → PE
LV → LE
LE maps to PE
Extent size is defined at VG creation:
vgcreate -s 8M vgdata /dev/sdb
Create LV using 100 extents:
lvcreate -l 100 -n lvdata vgdata
Create LV using size:
lvcreate -L 10G -n lvdata vgdata
Common commands:
pvs / pvdisplay
vgs / vgdisplay
lvs / lvdisplay
Add disk:
pvcreate /dev/sdc
vgextend vgdata /dev/sdc
Extend LV:
lvextend -L +10G /dev/vgdata/lvdata
Extend LV + filesystem:
lvextend -r -L +10G /dev/vgdata/lvdata
XFS:
xfs_growfs /data
ext4:
resize2fs /dev/vgdata/lvdata
Snapshot:
lvcreate -L 10G -s -n lvdata_snap /dev/vgdata/lvdata
Important:
LV size ≠ filesystem size
After extending LV, filesystem may also need to be extended.
XFS → can grow, cannot shrink
ext4 → can grow and shrink
LVM Snapshot ≠ Backup
Snapshot = point-in-time view using copy-on-write
Main Advantages of LVM:
1. Online expansion
→ Increase LV/filesystem size while mounted
→ No need to stop the application in many cases
2. Easy storage expansion
→ Add a new disk/PV to an existing VG
→ Then extend the LV
3. Flexible storage allocation
→ Allocate space to LVs as required
→ Don't need fixed partitions for everything
4. Combine multiple disks
→ Multiple PVs can be combined into one VG
5. Snapshots
→ Create point-in-time snapshots of LVs
→ Useful for backups/testing
======================================================================================================================================
Linux permissions control who can access files/directories.
Check:
ls -l
ls -ld /directory
Basic permissions:
r = read = 4
w = write = 2
x = execute = 1
Permission order:
Owner | Group | Others
Example:
-rwxr-xr--
Owner → rwx
Group → r-x
Others → r--
Numeric:
755 = rwxr-xr-x
644 = rw-r--r--
700 = rwx------
600 = rw-------
Commands:
chmod 755 file → change permissions
chown user:group file → change owner + group
chgrp group file → change group
Directory permissions:
r → list contents
w → create/delete/rename files
x → enter/traverse directory
SUID:
4xxx
Executable runs with file owner's effective UID.
Example:
chmod 4755 file
SGID:
2xxx
Executable → runs with file's group
Directory → new files inherit directory's group
Example:
chmod 2775 /shared
Sticky Bit:
1xxx
Users cannot normally delete/rename files belonging to other users
Example:
chmod 1777 /shared
Common example: /tmp
umask:
Controls default permissions for NEW files/directories.
Base:
File → 666
Directory → 777
umask 022:
File → 644
Directory → 755
umask 077:
File → 600
Directory → 700
ACL = Access Control List
Used when owner/group/others are not enough.
Allows specific users/groups to have specific permissions.
Check ACL:
getfacl file
Set ACL:
setfacl -m u:alice:rw file
Remove ACL:
setfacl -x u:alice file
If ls shows '+':
-rw-r-----+
↑
Extended ACL exists
Important difference:
chmod → basic permissions
ACL → granular permissions
umask → default permissions for new objects
SUID → owner privilege
SGID → group privilege / group inheritance
Sticky → restrict deletion in shared directories
======================================================================================================================
IP Address
→ Identifies a host/network interface.
Example: 192.168.1.10
Subnet
→ Defines network + host portion.
Example: 192.168.1.0/24
Gateway
→ Router used to reach other networks.
Check:
ip route
DNS
→ Converts hostname ↔ IP.
Check:
cat /etc/resolv.conf
dig google.com
nslookup google.com
Routing
→ Determines where packets should go.
Check:
ip route
PORTS
→ Identify services on a host.
22 → SSH
53 → DNS
80 → HTTP
443 → HTTPS
123 → NTP
389 → LDAP
636 → LDAPS
445 → SMB
2049 → NFS
3306 → MySQL
5432 → PostgreSQL
8080 → HTTP alternate
TCP
→ Connection-oriented + reliable.
→ SYN → SYN-ACK → ACK
UDP
→ Connectionless + no delivery guarantee.
ss
→ Check listening ports/connections.
ss -lntp
→ Listening TCP + process
ss -lunp
→ Listening UDP + process
ss -tunap
→ All TCP/UDP connections
ss -lntp | grep :22
→ Check SSH port
IP COMMANDS
ip addr
→ Show IP addresses/interfaces
ip link
→ Show interface status
ip route
→ Show routing table
PING
ping 192.168.1.1
→ Test IP connectivity using ICMP
Important:
Ping success does NOT mean application/port is working.
TRACEPATH / TRACEROUTE
tracepath 8.8.8.8
traceroute 8.8.8.8
→ Show network path/hops
DNS
dig google.com
nslookup google.com
→ Test DNS resolution
CURL
curl -v http://server:8080
curl -I https://server
→ Test application/HTTP/HTTPS connectivity
FIREWALL
Check:
firewall-cmd --state
firewall-cmd --get-active-zones
firewall-cmd --list-all
firewall-cmd --list-services
firewall-cmd --list-ports
Allow service:
firewall-cmd --permanent --add-service=http
firewall-cmd --reload
Allow single port:
firewall-cmd --permanent --add-port=8080/tcp
firewall-cmd --reload
Allow UDP:
firewall-cmd --permanent --add-port=53/udp
firewall-cmd --reload
Allow port range:
firewall-cmd --permanent --add-port=8000-8080/tcp
firewall-cmd --reload
Allow UDP range:
firewall-cmd --permanent --add-port=5000-6000/udp
firewall-cmd --reload
Remove port:
firewall-cmd --permanent --remove-port=8080/tcp
firewall-cmd --reload
TCPDUMP
→ Captures actual network packets.
Basic:
tcpdump -i any -n
Specific port:
tcpdump -i any -n port 8080
TCP port:
tcpdump -i any -n tcp port 443
UDP port:
tcpdump -i any -n udp port 53
Specific host:
tcpdump -i any -n host 192.168.1.10
Host + port:
tcpdump -i any -n host 192.168.1.10 and port 443
Source:
tcpdump -i any -n src host 192.168.1.10
Destination:
tcpdump -i any -n dst host 192.168.1.10
Save capture:
tcpdump -i any -n -w capture.pcap
Read capture:
tcpdump -r capture.pcap
Capture 100 packets:
tcpdump -i any -n -c 100
TCP FLAGS
[S] → SYN
[S.] → SYN-ACK
[.] → ACK
[R] → RST
[F.] → FIN-ACK
TCP TROUBLESHOOTING
SYN
→ Client is trying to connect.
SYN → SYN-ACK → ACK
→ TCP connection established.
SYN reaches server but no SYN-ACK
→ Check service, firewall, routing.
SYN → RST
→ Connection refused; often no service listening.
SYN repeatedly, no response
→ Timeout; investigate firewall/network/remote host.
Handshake succeeds but application fails
→ Check application/TLS/HTTP.
NETWORK TROUBLESHOOTING FLOW
ip addr
↓
ip route
↓
ping
↓
ss
↓
firewall-cmd
↓
curl
↓
tcpdump
↓
application logs
tcpdump:
tcpdump -i any -nn
tcpdump -i any -nn port 8080
tcpdump -i any -nn host 10.10.10.20
tcpdump -i any -nn host 10.10.10.20 and port 8080
tcpdump -i any -nn -w capture.pcap
============================================================================================================================
==================================================
Start: systemctl start rabbitmq-server
Stop: systemctl stop rabbitmq-server
Restart: systemctl restart rabbitmq-server
Status: systemctl status rabbitmq-server
Enable at boot: systemctl enable rabbitmq-server
Enable + start: systemctl enable --now rabbitmq-server
Check RabbitMQ alive: rabbitmq-diagnostics ping
Node status: rabbitmq-diagnostics status
Check listeners: rabbitmq-diagnostics listeners
Check alarms: rabbitmq-diagnostics alarms
Cluster status: rabbitmqctl cluster_status
Stop RabbitMQ application: rabbitmqctl stop_app
Start RabbitMQ application: rabbitmqctl start_app
Join cluster: rabbitmqctl join_cluster rabbit@rabbit1
Leave/reset node: rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app
IMPORTANT: reset removes the node's existing cluster membership/data. Use carefully.
Add user: rabbitmqctl add_user appuser 'StrongPassword'
List users: rabbitmqctl list_users
Change password: rabbitmqctl change_password appuser 'NewPassword'
Delete user: rabbitmqctl delete_user appuser
Administrator: rabbitmqctl set_user_tags appuser administrator
Monitoring: rabbitmqctl set_user_tags appuser monitoring
Remove tags: rabbitmqctl set_user_tags appuser
Create: rabbitmqctl add_vhost /production
List: rabbitmqctl list_vhosts
Delete: rabbitmqctl delete_vhost /production
Set permissions:
rabbitmqctl set_permissions -p /production appuser "." "." ".*"
Order:
configure write read
List permissions:
rabbitmqctl list_permissions -p /production
List user's permissions:
rabbitmqctl list_user_permissions appuser
Clear permissions:
rabbitmqctl clear_permissions -p /production appuser
List exchanges:
rabbitmqctl list_exchanges
Create using rabbitmqadmin:
rabbitmqadmin declare exchange
name=smsapp
type=topic
durable=true
Direct:
rabbitmqadmin declare exchange
name=my.direct
type=direct
durable=true
Fanout:
rabbitmqadmin declare exchange
name=my.fanout
type=fanout
durable=true
Headers:
rabbitmqadmin declare exchange
name=my.headers
type=headers
durable=true
List queues:
rabbitmqctl list_queues
Detailed queue information:
rabbitmqctl list_queues
name messages messages_ready messages_unacknowledged
Create queue:
rabbitmqadmin declare queue
name=sms.email
durable=true
Delete queue:
rabbitmqadmin delete queue name=sms.email
List bindings:
rabbitmqctl list_bindings
Create direct binding:
rabbitmqadmin declare binding
source=my.direct
destination=sms.email
destination_type=queue
routing_key=sms.email
Create topic binding:
rabbitmqadmin declare binding
source=smsapp
destination=sms.email
destination_type=queue
routing_key=sms.email
Topic wildcard:
routing_key='sms.*'
or:
routing_key='sms.#'
IMPORTANT:
Direct exchange: Exact match.
Topic exchange:
Publish:
rabbitmqadmin publish
exchange=smsapp
routing_key=sms.email
payload="Hello RabbitMQ"
Default exchange:
rabbitmqadmin publish
exchange=
routing_key=sms.email
payload="Hello"
List consumers:
rabbitmqctl list_consumers
Useful for checking:
List connections:
rabbitmqctl list_connections
Useful information:
List channels:
rabbitmqctl list_channels
Useful for:
Basic:
rabbitmqctl list_queues
Important:
rabbitmqctl list_queues
name messages messages_ready messages_unacknowledged
Meaning:
messages = total messages
messages_ready = waiting for consumers
messages_unacknowledged = delivered but not ACKed
List policies:
rabbitmqctl list_policies
Set policy example:
rabbitmqctl set_policy my-policy
"^sms."
'{"message-ttl":60000}'
--apply-to queues
Delete:
rabbitmqctl clear_policy my-policy
List parameters:
rabbitmqctl list_parameters
Used for things such as:
Check plugin:
rabbitmq-plugins list | grep federation
Enable:
rabbitmq-plugins enable rabbitmq_federation
Enable management federation UI/plugin if needed:
rabbitmq-plugins enable rabbitmq_federation_management
List parameters:
rabbitmqctl list_parameters
Federation status:
rabbitmqctl federation_status
Set upstream example:
rabbitmqctl set_parameter federation-upstream clusterA
'{"uri":"amqp://user:password@192.168.241.131:5672/%2f","prefetch-count":1000,"reconnect-delay":5}'
Federation policy example:
rabbitmqctl set_policy federate-sms
"^smsapp$"
'{"federation-upstream-set":"all"}'
--priority 10
--apply-to exchanges
List plugins:
rabbitmq-plugins list
Enable plugin:
rabbitmq-plugins enable PLUGIN_NAME
Disable:
rabbitmq-plugins disable PLUGIN_NAME
Check listening ports:
ss -lntp
Filter RabbitMQ ports:
ss -lntp | grep -E
'4369|5671|5672|15671|15672|25672'
Check RabbitMQ listeners:
rabbitmq-diagnostics listeners
Test AMQP port:
nc -zv 192.168.241.131 5672
Test AMQP TLS:
nc -zv 192.168.241.131 5671
Test Erlang distribution:
nc -zv 192.168.241.131 25672
Check EPMD:
epmd -names
AMQP TLS:
openssl s_client
-connect 192.168.241.131:5671
-showcerts
With hostname/SNI:
openssl s_client
-connect rabbitmq.example.com:5671
-servername rabbitmq.example.com
Follow RabbitMQ logs:
journalctl -u rabbitmq-server -f
Recent logs:
journalctl -u rabbitmq-server
Last 100 lines:
journalctl -u rabbitmq-server -n 100
Since today:
journalctl -u rabbitmq-server --since today
Check alarms:
rabbitmq-diagnostics alarms
Check node status:
rabbitmq-diagnostics status
Check disk:
df -h
Check memory:
free -h
Check processes:
ps -ef | grep rabbitmq
Create quorum queue:
rabbitmqadmin declare queue
name=sms.quorum
durable=true
arguments='{"x-queue-type":"quorum"}'
List queues and type:
rabbitmqctl list_queues name type
Create DLX:
rabbitmqadmin declare exchange
name=sms.dlx
type=direct
durable=true
Create DLQ:
rabbitmqadmin declare queue
name=sms.dlq
durable=true
Bind DLQ:
rabbitmqadmin declare binding
source=sms.dlx
destination=sms.dlq
destination_type=queue
routing_key=dead
Create main queue with DLX:
rabbitmqadmin declare queue
name=sms.email
durable=true
arguments='{"x-dead-letter-exchange":"sms.dlx","x-dead-letter-routing-key":"dead"}'
Retry queue:
rabbitmqadmin declare queue
name=sms.retry
durable=true
arguments='{"x-message-ttl":30000,"x-dead-letter-exchange":"smsapp","x-dead-letter-routing-key":"sms.email"}'
Meaning:
30,000 ms = 30 seconds TTL.
After expiry: retry queue ↓ main exchange ↓ main queue
List queues:
rabbitmqadmin list queues
List queue details:
rabbitmqadmin list queues
name messages messages_ready messages_unacknowledged
List exchanges:
rabbitmqadmin list exchanges
List bindings:
rabbitmqadmin list bindings
List connections:
rabbitmqadmin list connections
List consumers:
rabbitmqadmin list consumers
rabbitmqctl: Main CLI for RabbitMQ node administration.
Examples:
rabbitmqctl cluster_status rabbitmqctl list_users rabbitmqctl list_queues rabbitmqctl list_connections
rabbitmqadmin: Management HTTP/API-oriented administration tool.
Examples:
rabbitmqadmin declare queue rabbitmqadmin declare exchange rabbitmqadmin declare binding rabbitmqadmin publish
RabbitMQ problem:
rabbitmq-diagnostics ping
rabbitmq-diagnostics status
rabbitmqctl cluster_status
rabbitmq-diagnostics listeners
rabbitmq-diagnostics alarms
rabbitmqctl list_queues
name messages messages_ready messages_unacknowledged
rabbitmqctl list_consumers
rabbitmqctl list_connections
rabbitmqctl list_channels
journalctl -u rabbitmq-server -f
ss -lntp nc -zv
df -h
free -h
rabbitmq-diagnostics ping rabbitmq-diagnostics status rabbitmq-diagnostics listeners rabbitmq-diagnostics alarms
rabbitmqctl cluster_status
rabbitmqctl list_queues rabbitmqctl list_consumers rabbitmqctl list_connections rabbitmqctl list_channels rabbitmqctl list_bindings rabbitmqctl list_exchanges
rabbitmqctl list_users rabbitmqctl list_vhosts rabbitmqctl list_permissions -p /
rabbitmqctl list_policies rabbitmqctl list_parameters
rabbitmqctl federation_status
rabbitmq-plugins list
journalctl -u rabbitmq-server -f ss -lntp df -h free -h
HEALTH: ping → status → listeners → alarms
CLUSTER: cluster_status
QUEUE: list_queues
CONSUMER: list_consumers
CONNECTION: list_connections
CHANNEL: list_channels
ROUTING: list_exchanges → list_bindings
SECURITY: list_users → list_vhosts → list_permissions
FEDERATION: list_parameters → federation_status
NETWORK: ss → nc → epmd
LOGS: journalctl
RESOURCES: alarms → free → df
GOAL: Understand RabbitMQ internals, not memorize commands.
Producer ↓ Exchange ↓ routing key + binding Queue ↓ Consumer
Producer normally knows:
Producer does NOT need to know:
RabbitMQ admin defines queues + bindings.
Exchange = routing decision Queue = stores messages Consumer = processes messages
DIRECT: Exact routing-key match.
topic.sms → queue A topic.email → queue B
IMPORTANT: Direct exchange does NOT support * or # wildcards. "sms.*" is literal.
TOPIC: Pattern matching.
sms.* → sms.email sms.* → sms.email.audit ❌
sms.# → sms sms.# → sms.email sms.# → sms.email.audit
FANOUT: Ignores routing key. Sends to every bound queue.
HEADERS: Routes using message headers. x-match = all / any
DEFAULT EXCHANGE: Routing key is queue name. Producer can directly publish to a queue.
CLASSIC: Simple/lightweight queue.
QUORUM: Replicated HA queue. Uses Raft.
Example 3-member quorum:
Leader + Follower + Follower
3 members → tolerate 1 failure 5 members → tolerate 2 failures
IMPORTANT: RabbitMQ Cluster ≠ automatic queue replication.
Cluster shares topology/metadata. Queue replication depends on queue type/configuration.
STREAM: Designed for high-throughput stream-like workloads.
Cluster = multiple RabbitMQ nodes acting as one logical cluster.
Example:
rabbit1 rabbit2 rabbit3
Cluster shares metadata such as:
NOT every queue/message is automatically replicated.
Quorum queue provides queue-level replication.
Node failure: Quorum leader fails → remaining members elect new leader → clients connected to failed node lose connection → application must reconnect/recover
Cluster failover ≠ client connection failover.
5672 = AMQP 5671 = AMQP TLS 15672 = Management HTTP/API 15671 = Management HTTPS 4369 = EPMD 25672 = Erlang distribution (commonly)
Distribution port range can be configured, e.g.: 35672-35682
Always verify actual configuration before opening firewall ports.
Useful:
ss -lntp rabbitmq-diagnostics listeners epmd -names
Connection = TCP connection.
Channel = lightweight logical session inside connection.
One connection: ├── Channel 1 ├── Channel 2 └── Channel 3
Avoid creating a new TCP connection for every message/request.
Connection failure: → all channels on it are affected.
Channel failure: → other channels can continue.
Publisher Confirm: RabbitMQ confirms broker accepted the publish.
It does NOT mean: "Consumer successfully processed message."
Publisher Confirm ≠ Consumer ACK
Flow:
Producer ↓ publish RabbitMQ ↓ confirm Producer
Consumer: RabbitMQ ↓ message Consumer ↓ ACK RabbitMQ
ACK: Processing successful.
NACK/reject: Processing failed.
requeue=true: Message can be delivered again.
requeue=false: No same-queue requeue. Can go to DLX if configured.
BAD: NACK + requeue=true forever → redelivery loop.
Consumer count = number of workers.
Prefetch = outstanding message window per consumer.
Example:
5 consumers prefetch = 10
≈ 5 × 10 = 50 outstanding messages
Higher prefetch:
prefetch=1: Usually more balanced/fair.
Same queue + multiple consumers:
Queue
/ | \
C1 C2 C3
Messages are distributed among consumers.
Multiple queues with matching bindings:
Exchange ├── Queue A → Consumer └── Queue B → Consumer
Message can reach BOTH queues.
That is different from competing consumers.
Single consumer: Usually queue order.
Multiple consumers: No global processing-order guarantee.
Requeue/failure/prefetch can affect application-observed order.
Common properties:
delivery_mode content_type message_id correlation_id reply_to expiration headers
Routing key is NOT a message property. It is used by the exchange for routing.
DLX = Dead Letter EXCHANGE DLQ = Dead Letter QUEUE
DLX is NOT itself a queue.
Example:
Main Queue ↓ failure DLX ↓ DLQ
Typical reasons:
Main Exchange ↓ Main Queue ↓ failure Retry Exchange ↓ Retry Queue TTL = 30 sec ↓ expiry Main Exchange ↓ Main Queue
IMPORTANT: DLX ≠ retry automatically.
Need explicit retry design.
Avoid infinite retry. Use maximum attempts/business logic.
Durable queue: Queue survives broker restart.
Persistent message: Message is stored persistently according to RabbitMQ's persistence mechanisms.
For durable messaging: Use durable queue + persistent messages appropriately.
Quorum queues additionally provide replication.
Persistence ≠ replication.
AMQP transactions:
tx.select publish tx.commit or tx.rollback
Publisher confirms are generally preferred for performance-sensitive publishing.
RabbitMQ RPC commonly uses:
reply_to correlation_id
Request: Producer → request queue
Response: Consumer → reply queue
correlation_id identifies which response belongs to which request.
Authentication: WHO are you?
Authorization: WHAT are you allowed to do?
Local user:
rabbitmqctl add_user appuser password
Vhost:
rabbitmqctl add_vhost /production
Permissions:
rabbitmqctl set_permissions -p /production appuser "." "." ".*"
Permission order:
configure write read
User tags mainly affect management/UI capabilities.
LDAP can provide authentication.
RabbitMQ still needs authorization rules/permissions.
Can configure:
internal LDAP
LDAP: 389
LDAPS: 636
Keep a local break-glass admin when appropriate.
AMQP TLS: 5671
Important files:
CA certificate Server certificate Private key
Typical:
ssl_options.cacertfile ssl_options.certfile ssl_options.keyfile ssl_options.verify
mTLS:
verify_peer fail_if_no_peer_cert = true
"unable to get local issuer certificate" → client cannot build a trusted certificate chain.
curl -k: Disables certificate verification. Useful only for testing, NOT a security fix.
Hostname/SAN must also match.
Federation connects INDEPENDENT RabbitMQ brokers/clusters.
Cluster: One logical RabbitMQ cluster.
Federation: Independent brokers connected together.
Federation mental model:
Upstream ↓ Federation ↓ Downstream
Federated queue:
Upstream Queue ↓ Federation ↓ Downstream Queue ↓ Application Consumer
Federated exchange:
Upstream Exchange ↓ Federation ↓ Downstream Exchange ↓ Queue ↓ Consumer
Downstream federated queue behaves somewhat like a consumer of upstream.
SHOVEL:
Source Queue ↓ Shovel ↓ Destination Exchange/Queue
Easy memory:
FEDERATION = topology/following relationship SHOVEL = explicit message movement
Upstream definition: WHERE/HOW to connect.
Federation policy: WHAT object to federate.
Policy is configured on DOWNSTREAM.
If upstream is unavailable:
Federation link breaks ↓ Reconnect attempts ↓ Already transferred downstream messages remain downstream ↓ Messages existing only upstream remain upstream
Downstream can continue processing messages it already has.
rabbitmq-diagnostics ping → Is RabbitMQ responding?
rabbitmq-diagnostics status → What is happening on this node?
rabbitmqctl cluster_status → What does the cluster look like?
rabbitmq-diagnostics listeners → Which ports is RabbitMQ listening on?
rabbitmqctl list_queues name messages messages_ready messages_unacknowledged
READY: Waiting for delivery.
UNACKED: Delivered to consumer but ACK not received.
MOST IMPORTANT:
READY ↑ → consumer capacity/availability problem likely.
UNACKED ↑ → consumers received messages but aren't ACKing fast enough.
Publish rate: Producer → RabbitMQ
Delivery rate: RabbitMQ → Consumer
ACK rate: Consumer → RabbitMQ
Redelivery rate: Messages being delivered again.
Example:
Publish = 1000/s Consumer/ACK = 400/s
→ backlog grows.
Example:
Delivery = 500/s ACK = 100/s
→ UNACKED grows.
Always correlate: READY + UNACKED + rates.
rabbitmqctl list_connections
Connection = TCP.
Too many connections may indicate:
Do not create a new RabbitMQ TCP connection for every message.
rabbitmqctl list_channels
Channel = lightweight logical session inside a connection.
Useful when investigating:
rabbitmqctl list_consumers
Check:
Scenario:
READY = 50,000 UNACKED = 0 Consumers = 0
→ Consumer unavailable/configuration/connectivity problem likely.
Scenario:
READY = increasing Consumers exist
→ Consumer capacity may be lower than producer rate.
Scenario:
UNACKED = increasing
→ Consumer processing/ACK problem likely.
RabbitMQ protects itself from resource exhaustion.
Memory alarm: Memory usage exceeds configured threshold.
Disk alarm: Available disk space becomes too low.
Check:
rabbitmq-diagnostics alarms
Important:
RabbitMQ process UP ≠ RabbitMQ accepting publishes normally
A resource alarm can cause publishers to be blocked.
Message flow:
Queue ↓ Consumer receives ↓ UNACKED ↓ Application processes ↓ ACK ↓ RabbitMQ removes message
If consumer is slow: UNACKED ↑
If DB/API is slow: UNACKED ↑
If consumer hangs: UNACKED ↑
If consumer crashes before ACK: Message can be redelivered.
High prefetch can increase outstanding messages.
Problem: "RabbitMQ messages are stuck."
STEP 1: Is RabbitMQ alive?
rabbitmq-diagnostics ping
STEP 2: Node healthy?
rabbitmq-diagnostics status
STEP 3: Cluster healthy?
rabbitmqctl cluster_status
STEP 4: Check listeners/network.
rabbitmq-diagnostics listeners ss -lntp
STEP 5: Check queue.
messages_ready messages_unacknowledged
STEP 6: Check rates.
publish delivery ACK redelivery
STEP 7: Check consumers.
rabbitmqctl list_consumers
STEP 8: Check connections/channels.
rabbitmqctl list_connections rabbitmqctl list_channels
STEP 9: Check resource alarms.
rabbitmq-diagnostics alarms
STEP 10: Check consumer application logs, DB/API dependencies and processing time.
READY ↑ + UNACKED low → Consumers unavailable/slow capacity.
READY ↑ + UNACKED ↑ → Incoming rate > processing capacity.
READY low + UNACKED ↑ → Consumer processing/ACK problem.
Consumers = 0 → Consumer/application/config/network problem.
RabbitMQ UP + publishing blocked → Check resource alarms.
Consumer crashes before ACK → Unacked messages can be redelivered.
Redelivery ↑↑ → Check failures/requeue loops.
When troubleshooting RabbitMQ:
PRODUCER ↓ PUBLISH RATE ↓ EXCHANGE ↓ ROUTING/BINDING ↓ QUEUE ↓ READY ↓ DELIVERY ↓ CONSUMER ↓ PROCESSING ↓ ACK ↓ MESSAGE COMPLETE
At every stage ask:
"Where is the message currently stuck?"
rabbitmq-diagnostics ping rabbitmq-diagnostics status rabbitmq-diagnostics listeners rabbitmq-diagnostics alarms
rabbitmqctl cluster_status
rabbitmqctl list_queues rabbitmqctl list_consumers rabbitmqctl list_connections rabbitmqctl list_channels
ss -lntp epmd -names journalctl -u rabbitmq-server
RabbitMQ Cluster: Multiple nodes forming one logical cluster.
Quorum Queue: Replicated queue using Raft.
Exchange: Routes messages to queues.
Binding: Defines exchange → queue routing relationship.
Routing key: Input used by exchange for routing.
READY: Waiting for consumer delivery.
UNACKED: Delivered but not ACKed.
Prefetch: Limits outstanding messages per consumer.
Publisher Confirm: Broker accepted publish.
Consumer ACK: Consumer successfully processed/accepted responsibility.
DLX: Exchange receiving dead-lettered messages.
DLQ: Queue storing dead-lettered messages.
Federation: Connects independent RabbitMQ brokers/clusters.
Shovel: Moves messages explicitly from source to destination.
Authentication: Who are you?
Authorization: What can you do?
TLS: Protects communication.
DON'T RESTART FIRST.
CHECK:
HEALTH ↓ CLUSTER ↓ QUEUE ↓ READY / UNACKED ↓ RATES ↓ CONSUMERS ↓ CONNECTIONS ↓ ALARMS ↓ APPLICATION / DB / NETWORK
RABBITMQ / RHEL / KUBERNETES / DEVOPS INTERVIEW STUDY
=============================================================
PURPOSE
-------
This is the consolidated study summary of the RabbitMQ topics already learned, followed by the pending topics. Use this as the context for continuing the study later.
=============================================================
PART 1 — RABBITMQ FUNDAMENTALS
=============================================================
1. RabbitMQ is a message broker.
Main flow:
Producer -> Exchange -> Binding -> Queue -> Consumer
2. Producer normally publishes to an exchange, not directly to a queue.
3. Exchange and queue are separate objects.
Creating an exchange does NOT create a queue inside it.
A binding connects an exchange to a queue.
4. Routing key:
- A value supplied by the publisher.
- Direct exchange: exact match.
- Topic exchange: pattern matching through bindings.
- Fanout: routing key is effectively ignored.
- Headers: message headers determine routing.
- Routing key is NOT authentication.
5. Direct exchange:
Exact routing-key match.
Multiple queues can have the same routing key, so one message can be routed to multiple queues.
6. Topic exchange:
- * = exactly one word.
- # = zero or more words.
Example:
binding sms.* matches sms.otp and sms.nonotp
binding sms.# can match sms, sms.otp, sms.otp.high
7. Fanout exchange:
Sends a message to all queues bound to that exchange.
It does NOT broadcast to every queue in RabbitMQ.
8. Headers exchange:
Routes using message headers.
x-match=all -> all matching headers.
x-match=any -> any matching header.
9. Default exchange:
- Name is empty string "".
- Direct exchange.
- Every queue is automatically bound to it using the queue name as routing key.
Example:
exchange="" + routing_key="nonotp" -> queue nonotp
A custom exchange with no binding does NOT fall back to the default exchange.
10. Vhost:
Logical namespace/isolation boundary.
Contains exchanges, queues, bindings and permissions.
/ is the default vhost.
Vhost is not another RabbitMQ server.
=============================================================
PART 2 — QUEUES, DURABILITY AND DELIVERY
=============================================================
11. Durable queue:
Queue definition survives broker restart.
Durable does NOT mean replicated.
12. Persistent message:
Message is marked for durable storage.
For restart durability, normally use durable queue + persistent message.
13. Persistence/durability != HA/replication.
14. Queue types:
- Classic
- Quorum
- Stream
15. Classic queue:
Traditional queue.
Normally hosted on one node and not replicated by default.
Classic queue itself is NOT deprecated.
16. Classic queue mirroring:
Old HA mechanism.
Deprecated in RabbitMQ 3.9.
Removed in RabbitMQ 4.0.
17. Quorum queue:
Replicated queue using Raft consensus.
Usually has one leader and followers.
Each quorum queue has its own Raft group.
There is no single cluster-wide queue leader.
18. Quorum majority:
3 members -> 2/3 majority is enough to continue.
If only 1/3 remains, quorum is lost and safe progress cannot continue.
One-member quorum queue is possible but provides no HA.
19. Quorum leader failure:
Remaining majority elects a new leader.
Application does not need to track the queue leader.
20. Application connection and queue leader are different concepts.
Application can connect to Node3 while the queue leader is Node1.
21. Cluster membership does NOT automatically replicate every queue.
Classic queue data can remain on one node.
22. Classic queue on Node1:
Node2/3 know queue metadata but do not automatically have copies of the messages.
If Node1 restarts, durable/persistent data can normally recover from its disk.
Permanent loss of Node1 can mean loss/unavailability of that classic queue's data.
=============================================================
PART 3 — ACK, NACK, PREFETCH, CONNECTIONS
=============================================================
23. ACK:
Consumer tells RabbitMQ that the delivery was successfully handled.
The message is no longer outstanding for redelivery.
24. NACK:
Consumer explicitly tells RabbitMQ it could not handle the message.
requeue=true -> message can be requeued.
requeue=false -> message is not requeued; with DLX it can be dead-lettered.
25. No ACK is NOT the same as NACK.
If consumer stays connected, message remains unacknowledged.
If channel/connection closes before ACK, message can be requeued/redelivered.
26. RabbitMQ knows a message was delivered because it sent the AMQP delivery.
ACK is the consumer's confirmation of successful handling, not merely confirmation that TCP received a packet.
27. Redelivery:
Same message can be delivered again after requeue or consumer/channel failure.
Applications should be idempotent/deduplicate when duplicate business operations are possible.
RabbitMQ generally provides at-least-once style delivery, not exactly-once business processing.
28. Prefetch:
Controls how many messages can be outstanding/unacknowledged for a consumer under the applicable QoS setting.
Example prefetch=10:
RabbitMQ can have up to about 10 outstanding messages.
If 5 are ACKed, 5 slots become available and RabbitMQ can send 5 more.
Prefetch is NOT messages/second and NOT necessarily a batch size.
RabbitMQ does not wait for all 10 ACKs before replacing freed slots.
29. If prefetch=10 and consumer ACKs nothing:
10 messages remain unacknowledged.
RabbitMQ does not simply send another 10 while the same consumer remains active.
30. If consumer dies before ACK:
Unacknowledged messages can be requeued and delivered again.
31. Connection:
Actual TCP/network connection to RabbitMQ.
32. Channel:
Lightweight logical AMQP session inside a connection.
One connection can have multiple channels.
Connection failure affects its channels.
A channel can fail while the TCP connection remains available for other channels.
33. Publisher confirms:
Broker confirmation to publisher that a publish was accepted according to RabbitMQ's confirmation semantics.
Different from consumer ACK.
=============================================================
PART 4 — TTL, DLX, DLQ AND RETRY
=============================================================
34. TTL:
Time-to-live for messages.
Can be configured at queue level or per message.
No default message TTL.
Effective expiration can be affected by the shorter applicable TTL.
Physical removal may occur when RabbitMQ processes expiration, not necessarily exactly at the expiry timestamp.
35. DLX:
Dead Letter Exchange.
Not automatically enabled.
Can receive messages that are dead-lettered.
36. DLQ:
Dead Letter Queue.
Usually a normal queue bound to a DLX.
RabbitMQ does not automatically consume the DLQ.
37. Dead-letter causes include:
- NACK/reject with requeue=false
- TTL expiration
- Queue length limit
- Other configured dead-letter conditions
38. Typical flow:
Original queue -> DLX -> binding -> DLQ -> separate consumer/process
39. Retry:
Can be implemented using requeue, retry queues with TTL+DLX, or application logic.
requeue=true can cause retry storms.
Temporary failures -> retry.
Permanent/repeated failures -> DLQ.
=============================================================
PART 5 — CLUSTERING, HA AND NETWORK PARTITIONS
=============================================================
40. RabbitMQ cluster:
Multiple RabbitMQ nodes form one logical cluster.
3 nodes are common for production HA but 3 is NOT a technical requirement.
41. Cluster membership:
Nodes use Erlang/RabbitMQ distribution and the Erlang cookie.
Common cookie path:
/var/lib/rabbitmq/.erlang.cookie
42. Common cluster commands:
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@node1
rabbitmqctl start_app
rabbitmqctl cluster_status
43. rabbitmqctl vs rabbitmqadmin:
rabbitmqctl -> node/CLI administration.
rabbitmqadmin -> Management HTTP API based administration.
44. rabbitmqctl reset is destructive.
Do not casually run it in production.
45. Network partition / split brain:
Nodes are alive but cannot communicate.
In a 3-member quorum queue, a 2-node majority can continue while the isolated 1-node minority cannot safely make progress.
Quorum prevents independent minority writes.
46. Application may still have a TCP connection to an isolated minority node.
It may not immediately know that the node is a minority.
Symptoms can be failed operations, missing publisher confirms, timeouts or connection recovery.
Exact timeout/recovery depends on client/application configuration.
47. Publisher confirms help determine whether a publish was accepted.
Missing confirmation does NOT always prove message loss because the broker may have accepted the message while the confirmation was lost.
Blind retry can create duplicates.
Use idempotency/deduplication for business operations such as SMS.
=============================================================
PART 6 — FEDERATION AND SHOVEL
=============================================================
48. Federation:
Connects independent RabbitMQ brokers/clusters.
Does NOT merge them into one cluster.
Useful for multi-DC/region or independent environments.
49. Federated exchange:
Upstream exchange -> federation -> downstream exchange -> normal bindings -> queue -> consumer.
50. Federated queue:
Downstream side can obtain messages from an upstream queue.
It is NOT permanent queue replication and NOT quorum HA.
51. Shovel:
Long-running message transport mechanism.
Consumes from configured source and publishes to configured destination.
Useful for migration, integration and controlled one-way message movement.
Source/destination can be queue/exchange depending configuration.
52. Federation vs Shovel:
Federation -> broker/resource-level ongoing relationship between independent environments.
Shovel -> explicit source-to-destination message transport.
Neither is HA/replication.
=============================================================
PART 7 — SECURITY
=============================================================
53. Authentication:
Answers: Who are you?
RabbitMQ can use local users/passwords or LDAP.
54. Authorization:
Answers: What can you access/do?
Controlled through vhosts and permissions.
55. RabbitMQ permissions:
- configure
- write
- read
56. Useful commands:
rabbitmqctl list_users
rabbitmqctl list_vhosts
rabbitmqctl list_permissions -p /
rabbitmqctl list_user_permissions <user>
57. Example:
rabbitmqctl set_permissions -p /sms smsapp ".*" ".*" ".*"
This grants broad permissions; production should use least privilege.
58. TLS:
5672 -> normal AMQP
5671 -> AMQP over TLS
15672 -> Management HTTP
15671 -> Management HTTPS
59. TLS provides:
Confidentiality
Integrity
Server authentication
60. CA:
Certificate Authority signs certificates.
Java truststore normally contains trusted CA certificates.
61. Truststore:
"Who do I trust?"
Used to verify RabbitMQ server certificate.
62. Keystore:
"Who am I?"
Contains client certificate/private key when client identity is needed, such as mTLS.
63. Normal one-way TLS:
RabbitMQ presents server certificate.
Java client verifies it using truststore.
Client certificate is not normally required.
64. mTLS:
Both sides authenticate using certificates.
RabbitMQ can require client certificates.
Java client uses keystore for client identity and truststore for trusted CA.
mTLS does not replace RabbitMQ authorization.
65. Example RabbitMQ TLS settings:
listeners.ssl.default = 5671
ssl_options.cacertfile = /etc/rabbitmq/certs/ca.pem
ssl_options.certfile = /etc/rabbitmq/certs/server.pem
ssl_options.keyfile = /etc/rabbitmq/certs/server.key
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
66. TLS troubleshooting:
Network -> 5671 -> certificate chain/truststore -> hostname/SAN -> TLS versions/ciphers -> mTLS client cert -> RabbitMQ authentication -> vhost/permissions.
67. LDAP:
RabbitMQ 4.x uses rabbitmq_auth_backend_ldap plugin.
Enable:
rabbitmq-plugins enable rabbitmq_auth_backend_ldap
68. LDAP can authenticate while RabbitMQ internal backend handles authorization:
auth_backends.1.authn = ldap
auth_backends.1.authz = internal
69. LDAP authorization can use:
vhost_access_query
resource_access_query
topic_access_query
tag_queries
Exact advanced.config syntax should be verified against the target RabbitMQ version before production.
=============================================================
PART 8 — BACKUP, RESTORE AND MIGRATION
=============================================================
70. Definitions backup:
rabbitmqctl export_definitions /backup/rabbitmq-definitions.json
71. Restore:
rabbitmqctl import_definitions /backup/rabbitmq-definitions.json
72. Definitions include:
vhosts, users, permissions, exchanges, queues, bindings, policies, parameters, etc.
73. Definitions backup does NOT contain queued message bodies.
74. There is no general native "dump one queue's messages and restore it" command.
Message-level export can be done with application logic, Management API get/requeue workflows, shovel, or consume/re-publish strategies depending requirements.
75. Do not casually copy a live /var/lib/rabbitmq directory as a consistent backup.
For message-store/data backup, follow the RabbitMQ-supported procedure and normally stop the node for a consistent filesystem-level backup.
76. Blue-Green/fresh-cluster migration:
- Move application to DR
- Stop/quiesce producers
- Verify production queues are drained
- Export definitions
- Backup configuration/certs
- Stop old cluster
- Provision clean RabbitMQ 4.x + compatible Erlang
- Build new cluster
- Import definitions
- Validate
- Switch application back
77. This is a fresh-cluster/Blue-Green migration, NOT a rolling upgrade.
78. Rolling upgrade:
Only use a source->target version path explicitly supported by RabbitMQ.
Mixed versions may temporarily coexist when supported.
Upgrade one node, verify, continue.
Feature flags should be enabled only according to the supported upgrade process.
=============================================================
PART 9 — START/STOP, HEALTH CHECKS AND DATA DIRECTORY
=============================================================
79. Useful commands:
systemctl status rabbitmq-server
systemctl stop rabbitmq-server
systemctl restart rabbitmq-server
rabbitmqctl stop
rabbitmq-diagnostics status
rabbitmq-diagnostics ping
rabbitmq-diagnostics alarms
rabbitmq-diagnostics listeners
rabbitmqctl cluster_status
80. Queue/consumer checks:
rabbitmqctl list_queues name type messages messages_ready messages_unacknowledged
rabbitmqctl list_connections
rabbitmqctl list_channels
rabbitmqctl list_consumers
81. RabbitMQ data directory commonly:
/var/lib/rabbitmq
82. Custom data directory can be configured using:
RABBITMQ_MNESIA_BASE=/app/rabbitmq
usually in /etc/rabbitmq/rabbitmq-env.conf
83. User's lab:
NODENAME=rabbit@node8
RABBITMQ_MNESIA_BASE=/app/rabbitmq
84. Active Mnesia directory can be checked:
rabbitmqctl eval 'rabbit_mnesia:dir().'
85. Important:
Stop RabbitMQ before moving/copying node data.
Preserve ownership/permissions.
Do not blindly merge Mnesia databases.
Do not delete the old directory until the new setup is validated.
Custom paths on RHEL/Rocky may require SELinux context handling.
Never share one RabbitMQ data directory between multiple RabbitMQ nodes.
=============================================================
PART 10 — MEMORY, DISK, CPU AND PERFORMANCE
=============================================================
86. RabbitMQ uses both memory and disk.
Do not simplify this to "messages are always in RAM and then move to disk."
Internal storage behavior is more sophisticated.
87. Memory alarm:
RabbitMQ has a memory high-watermark/protection mechanism.
When memory pressure reaches the threshold, RabbitMQ can raise a memory alarm and apply publisher flow control.
88. Commands:
rabbitmq-diagnostics alarms
rabbitmq-diagnostics memory_breakdown
free -h
top
89. free -h is OS-wide, not RabbitMQ-specific.
It includes filesystem cache, kernel memory and other processes.
90. Disk alarm:
RabbitMQ protects itself when available disk space becomes too low.
Publishers can be blocked/slowed.
91. Commands:
df -h
df -h /app/rabbitmq
rabbitmq-diagnostics alarms
92. Disk capacity vs disk performance:
df -h -> how much space is available
iostat -xz 1 -> disk I/O/performance
93. CPU:
top
mpstat -P ALL 1
94. Network:
sar -n DEV 1
ip -s link
tcpdump
95. Performance depends on:
CPU
memory
disk I/O
network
message size
message rate
persistence
publisher confirms
queue type
quorum replication
consumers
prefetch
downstream dependencies
96. Producer rate vs consumer rate:
If producer > consumer, backlog grows.
backlog growth approximately = producer rate - consumer rate.
97. Example:
Producer 10,000 msg/sec
Consumer 7,000 msg/sec
Backlog grows about 3,000 msg/sec.
98. Message size matters:
200 KB × 5,000 msg/sec ≈ 1 GB/sec raw payload.
This is before protocol/replication/storage overhead.
99. Connection capacity:
No universal connections/sec limit.
Concurrent connections and connection-establishment rate are different.
Reuse long-lived connections.
OS file descriptors, CPU, RAM, TLS and workload affect capacity.
100. User's example hardware:
8 GB RAM, 500 GB disk, 8 CPU cores, 10 Gbps NIC.
Previous rough planning estimate: 5k–10k concurrent connections per node as an initial planning range, not a RabbitMQ limit; benchmark actual workload.
101. 80 million messages:
Storage depends on average message size.
80M × 200 KB ≈ 16 TB raw payload.
Message count alone is insufficient for capacity planning.
=============================================================
PART 11 — QUEUE TROUBLESHOOTING AND PERFORMANCE
=============================================================
102. messages_ready:
Messages waiting to be delivered.
103. messages_unacknowledged:
Messages delivered to consumers but not yet ACKed.
104. High ready + low unacked:
Consumers absent, slow, disconnected or unable to keep up.
105. Low ready + high unacked:
Consumer has messages but is slow/stuck/not ACKing.
106. Both zero:
Queue is empty; could be healthy or producer may not be publishing.
107. Consumer throughput:
Depends on number of consumers and processing time, but downstream DB/API/workload often becomes the real bottleneck.
108. Example:
100 ms/message ≈ 10 msg/sec/consumer.
10 consumers ≈ 100 msg/sec in a simple idealized case.
109. High prefetch:
Can produce many unacknowledged messages, memory pressure and uneven work distribution.
Do not blindly increase it.
110. Low prefetch:
Can reduce pipeline utilization if consumer processing has latency.
111. Good tuning method:
Measure consumer processing time, ready/unacked counts and throughput.
Adjust prefetch gradually while monitoring throughput, latency, memory and fairness.
112. Message-store optimization:
Find why messages accumulate.
Avoid unnecessary huge payloads.
Consider external object/file storage plus a reference in RabbitMQ for very large payloads.
Optimize consumers instead of simply increasing storage.
113. Do not purge production queues without confirming business impact and approval.
=============================================================
PART 12 — PUBLISHER THROUGHPUT
=============================================================
114. Publisher throughput can be limited by:
CPU
network
disk I/O
message size
persistence
publisher confirms
quorum replication
connections/channels
memory/disk alarms
downstream/storage pressure
115. Use long-lived connections.
Do not create a connection per message.
116. Publisher confirms improve delivery assurance but add overhead.
117. Quorum queues add replication/consensus/storage/network work compared with a non-replicated classic queue.
118. If publishing becomes slow:
Check CPU, disk I/O, network, alarms, message rate/size, confirms, queue type and client behavior.
=============================================================
PART 13 — CONNECTION AND CHANNEL TROUBLESHOOTING
=============================================================
119. Connection troubleshooting sequence:
DNS/hostname
-> network
-> firewall
-> port
-> RabbitMQ listener
-> TLS
-> authentication
-> vhost
-> authorization
-> resource
120. Commands:
getent hosts rabbit1
nslookup rabbit1
nc -vz rabbit1 5672
nc -vz rabbit1 5671
rabbitmq-diagnostics listeners
ss -lntp
firewall-cmd --list-all
rabbitmqctl list_connections
121. Heartbeats:
AMQP connection-liveness mechanism.
Helps detect failed/inactive connections.
Not an application health check.
122. Cluster networking is separate from application connectivity.
Application -> RabbitMQ on 5672/5671 is one path.
RabbitMQ node-to-node communication is another.
123. Channel troubleshooting:
Check list_channels and logs.
Look for PRECONDITION_FAILED, ACCESS_REFUSED, NOT_FOUND.
Common causes include declaring a queue/exchange with conflicting properties, invalid operations and permission issues.
124. Important distinction:
Connection failure affects channels belonging to that connection.
Channel failure does not necessarily bring down the whole TCP connection.
=============================================================
PART 14 — THREAD DUMPS
=============================================================
125. Thread dump:
Snapshot of Java application threads at a moment in time.
126. Important because:
RabbitMQ itself is Erlang/BEAM.
Java thread dumps are primarily for Tomcat/Java publisher/consumer applications.
127. Useful commands:
jps -l
ps -ef | grep java
jstack <PID> > thread_dump.txt
kill -3 <PID>
128. Taking multiple dumps:
Example:
dump 1 at t=0
dump 2 at t=5 sec
dump 3 at t=10 sec
If the same threads remain stuck in the same place, it is stronger evidence of a persistent problem.
129. Thread states:
RUNNABLE -> running or ready to run; does not always mean actively executing on CPU.
BLOCKED -> waiting to acquire a Java monitor lock held by another thread.
WAITING -> waiting indefinitely for an event/notification.
TIMED_WAITING -> waiting with a timeout.
130. Lock contention:
One thread holds a lock and another waits.
May be temporary.
131. Deadlock:
Circular lock dependency.
Example:
Thread A holds Lock1 and wants Lock2.
Thread B holds Lock2 and wants Lock1.
JVM thread dump may report a Java-level deadlock.
132. RabbitMQ connection to Java troubleshooting:
RabbitMQ can be healthy while Java consumer threads are blocked on DB, network, locks or other work.
This can delay ACKs and increase unacknowledged messages.
=============================================================
PART 15 — HEAP DUMPS
=============================================================
133. Thread dump vs heap dump:
Thread dump -> what threads are doing.
Heap dump -> what Java objects are consuming/retaining memory.
134. Java heap:
Memory area where Java objects are allocated.
135. Java memory leak:
Application unintentionally retains references to objects it no longer needs, preventing GC from reclaiming them.
136. Useful commands:
jps -l
jmap -dump:live,format=b,file=/tmp/heap.hprof <PID>
jcmd <PID> GC.heap_dump /tmp/heap.hprof
137. Heap dump analysis asks:
Which classes/objects consume the most memory?
What is retaining them?
Why can't GC reclaim them?
138. Tools:
Eclipse MAT and other heap-analysis tools.
139. RabbitMQ relation:
Java consumer memory leak -> GC pressure -> application slows -> ACKs delayed -> RabbitMQ unacked count increases.
RabbitMQ may not be the root cause.
=============================================================
PART 16 — TROUBLESHOOTING MINDSET
=============================================================
140. Never immediately restart/purge when a problem appears.
141. Use layered troubleshooting:
Application
-> connection
-> channel
-> RabbitMQ node
-> cluster
-> queue
-> consumer/publisher
-> CPU/memory/disk/network
-> downstream dependencies
142. Always correlate:
- queue ready/unacked
- producer rate
- consumer rate
- CPU
- memory
- disk capacity
- disk I/O
- network
- application logs
- RabbitMQ logs
143. A RabbitMQ symptom does not necessarily mean RabbitMQ is the root cause.
Example:
Java DB lock -> consumer stuck -> no ACK -> RabbitMQ unacked grows.
=============================================================
PART 17 — PENDING TOPICS
=============================================================
A. Java/JVM/RabbitMQ troubleshooting
------------------------------------
1. GC troubleshooting
2. OutOfMemoryError
3. GC logs and GC pauses
4. Heap dump practical analysis
5. RabbitMQ support bundle
6. Log-based RCA
7. RCA methodology
8. Complete production incident workflow
9. Deep queue troubleshooting scenarios
10. Deep consumer troubleshooting scenarios
11. Deep publisher troubleshooting scenarios
B. RabbitMQ operations/scripts
------------------------------
12. Health-check shell script
13. Monitoring script
14. Start/stop/restart script
15. Maintenance scripts
16. Automated queue/consumer checks
C. RHEL/Linux
-------------
17. RHEL networking fundamentals
18. Ports and firewall
19. DNS/hostname troubleshooting
20. Routing/network troubleshooting
21. Storage troubleshooting
22. LVM
23. Filesystem troubleshooting
24. Disk I/O troubleshooting
D. Other JD topics
------------------
25. Caching
26. IMDG
27. NoSQL
28. Kubernetes architecture
29. Pod lifecycle
30. Deployments
31. Kubernetes + RabbitMQ
32. CI/CD
33. DevOps concepts
E. Final interview preparation
------------------------------
34. RabbitMQ scenario-based troubleshooting
35. Failure/HA scenarios
36. Security/TLS/LDAP scenarios
37. Backup/migration/upgrade scenarios
38. RHEL troubleshooting scenarios
39. Kubernetes/RabbitMQ scenarios
40. Full mock interview
41. RCA-based interview questions
=============================================================
CONTINUE FROM HERE
=============================================================
Next topic:
GC troubleshooting + OutOfMemoryError
When continuing, teach each topic:
1. Simple concept
2. Why it happens
3. Practical commands
4. Production example
5. Interview-ready answer
6. Common mistakes/traps
7. Then ask/continue to the next topic only after the user says OK.
=============================================================
COPY-PASTE CONTINUATION PROMPT
=============================================================
I am preparing for a RabbitMQ/RHEL/Kubernetes/DevOps interview based on a JD involving RabbitMQ administration, clustering, HA, federation, shovels, TLS/LDAP, troubleshooting, RCA, shell scripting, RHEL networking/storage/LVM, caching, IMDG, NoSQL, Kubernetes, CI/CD and DevOps.
Use the following study progress as context:
COMPLETED RABBITMQ:
- RabbitMQ architecture and Producer -> Exchange -> Binding -> Queue -> Consumer
- Direct, Topic, Fanout, Headers exchanges
- Default exchange
- Routing keys and bindings
- Vhosts
- Queues
- Durable queues
- Persistent messages
- ACK/NACK/manual/auto ACK
- Redelivery and at-least-once behavior
- Prefetch/QoS
- Connections vs channels
- Publisher confirms
- TTL
- DLX/DLQ
- Retry
- RabbitMQ clustering
- Cluster vs replication
- Quorum queues
- Raft, leader/followers, majority and elections
- Classic vs quorum
- Classic queue mirroring deprecation/removal
- Federation
- Federated exchange/queue
- Shovel
- Federation vs Shovel
- HA
- Node failures
- Quorum failures
- Network partitions/split-brain
- Application connection recovery concepts
- Authentication
- Authorization
- Users/vhosts/permissions
- configure/write/read permissions
- TLS/SSL
- Certificates
- Truststore vs keystore
- mTLS
- LDAP authentication and internal authorization
- Backup/restore
- Definitions export/import
- Message-store/data backup concepts
- Blue-Green migration
- Rolling upgrade concepts and compatibility
- Start/stop/restart
- Health checks
- RabbitMQ data directory/Mnesia migration
- Memory alarms
- Disk alarms
- CPU/resource troubleshooting
- Network troubleshooting
- Performance troubleshooting
- Queue backlog troubleshooting
- Consumer throughput
- Publisher throughput
- Message-store optimization
- Queue performance tuning
- Prefetch real-world behavior
- Connection troubleshooting
- Channel troubleshooting
- Thread dumps
- Java thread states
- Lock contention/deadlock
- Heap dumps and Java memory leaks
IMPORTANT PREFETCH UNDERSTANDING:
If prefetch=10 and one consumer receives 10 messages, RabbitMQ allows up to 10 outstanding/unacknowledged messages for that consumer. It does NOT wait for all 10 ACKs before sending more. If 5 ACKs arrive, 5 slots become available and RabbitMQ can send 5 more. If no ACKs arrive and the consumer remains alive, RabbitMQ does not simply send another 10. If the consumer/channel dies before ACK, the messages can be requeued/redelivered.
IMPORTANT ACK UNDERSTANDING:
RabbitMQ knows it delivered a message because it sent the AMQP delivery. ACK is the consumer's confirmation that processing/handling succeeded. No ACK does not equal NACK. If the consumer dies before ACK, the message can be redelivered. Consumers should handle duplicates/idempotency.
IMPORTANT TROUBLESHOOTING MODEL:
Application -> connection -> channel -> RabbitMQ node -> cluster -> queue -> consumer/publisher -> CPU/memory/disk/network -> downstream dependencies.
PENDING TOPICS:
1. GC troubleshooting
2. OutOfMemoryError
3. GC logs and GC pauses
4. Practical heap dump analysis
5. RabbitMQ support bundle
6. Log-based RCA
7. RCA methodology
8. Complete production incident workflow
9. Deep queue troubleshooting scenarios
10. Deep consumer troubleshooting scenarios
11. Deep publisher troubleshooting scenarios
12. Health-check shell script
13. Monitoring script
14. Start/stop/restart script
15. Maintenance scripts
16. Automated queue/consumer checks
17. RHEL networking
18. Ports/firewall
19. DNS/hostname
20. Routing/network troubleshooting
21. Storage troubleshooting
22. LVM
23. Filesystem troubleshooting
24. Disk I/O
25. Caching
26. IMDG
27. NoSQL
28. Kubernetes architecture
29. Pod lifecycle
30. Deployments
31. Kubernetes + RabbitMQ
32. CI/CD
33. DevOps
34. RabbitMQ scenario-based troubleshooting
35. HA/failure scenarios
36. Security/TLS/LDAP scenarios
37. Backup/migration/upgrade scenarios
38. RHEL troubleshooting scenarios
39. Kubernetes/RabbitMQ scenarios
40. Full mock interview
41. RCA-based interview questions
Continue from:
GC troubleshooting + OutOfMemoryError
Teaching style:
- Explain simply first.
- Use practical examples.
- Use RabbitMQ/Tomcat examples where useful.
- Explain WHY, not just commands.
- Give important Linux/RabbitMQ commands.
- Clearly distinguish RabbitMQ problems from Java application problems.
- Give an interview-ready answer after each major topic.
- Point out common interview traps/corrections.
- Do not skip fundamentals.
- Proceed topic by topic and wait for my "OK" before moving to the next topic.
KUBERNETES SECURITYCONTEXT — CKA REVISION
1. WHAT IS SECURITYCONTEXT?
---------------------------
SecurityContext defines security-related settings for a Pod or Container.
It controls:
- Linux user/group
- Root vs non-root
- File permissions
- Linux capabilities
- Privilege escalation
- Privileged mode
- Read-only filesystem
- Seccomp
- SELinux
Two levels:
- Pod-level securityContext → applies to Pod/containers by default
- Container-level securityContext → applies specifically to that container
If a setting exists at both Pod and Container level, container-level setting takes precedence.
2. POD-LEVEL SECURITYCONTEXT
----------------------------
Example:
spec:
securityContext:
runAsUser: 1000
runAsGroup: 3000
runAsNonRoot: true
fsGroup: 2000
3. runAsUser
------------
Specifies the Linux UID that the container process runs as.
securityContext:
runAsUser: 1000
Meaning:
- Process runs as UID 1000
- Instead of root (UID 0)
Purpose:
- Avoid running applications as root
- Improve security
4. runAsGroup
-------------
Specifies the primary Linux GID.
securityContext:
runAsUser: 1000
runAsGroup: 3000
Meaning:
- UID = 1000
- Primary GID = 3000
5. runAsNonRoot
---------------
Ensures the container does NOT run as root.
securityContext:
runAsNonRoot: true
Important:
runAsNonRoot: true does NOT mean UID 1000.
It means:
"The process must not run as UID 0."
Difference:
runAsUser → "Run as this UID"
runAsNonRoot → "Do NOT run as root"
6. fsGroup
----------
Defines the group ownership/access behavior for files associated with volumes mounted into the Pod.
Example:
spec:
securityContext:
fsGroup: 2000
Think:
Application
↓
Volume
↓
fsGroup
↓
Group-based access
Useful when an application needs group-based access to mounted volumes.
7. fsGroupChangePolicy
----------------------
Controls when Kubernetes changes volume ownership/permissions for fsGroup.
Values:
- Always
- OnRootMismatch
Always:
Always apply the ownership/permission changes.
OnRootMismatch:
Apply them only when the volume root does not already have the expected ownership/permissions.
8. supplementalGroups
---------------------
Adds additional Linux groups to the container process.
Example:
securityContext:
supplementalGroups:
- 4000
- 5000
Think:
Primary group
+
Supplementary groups
9. privileged
-------------
Runs a container in privileged mode.
Example:
securityContext:
privileged: true
Meaning:
- Gives the container highly elevated access
- Can access host devices/kernel capabilities
Generally:
privileged: false
Remember:
Privileged = highly elevated container access
10. CAPABILITIES
----------------
Linux capabilities split root privileges into individual permissions.
Example:
securityContext:
capabilities:
add:
- NET_ADMIN
drop:
- ALL
Can:
- ADD capabilities
- DROP capabilities
Example:
capabilities:
drop:
- ALL
Meaning:
Remove all capabilities.
Example:
capabilities:
add:
- NET_ADMIN
Meaning:
Give NET_ADMIN capability.
Security best practice:
Drop unnecessary capabilities and add only what is required.
11. allowPrivilegeEscalation
----------------------------
Controls whether a process can gain more privileges than its parent process.
Example:
securityContext:
allowPrivilegeEscalation: false
Meaning:
Do not allow the process to gain additional privileges.
For restricted security configurations:
allowPrivilegeEscalation: false
12. readOnlyRootFilesystem
--------------------------
Makes the container's root filesystem read-only.
Example:
securityContext:
readOnlyRootFilesystem: true
Meaning:
Container cannot write to its root filesystem.
If application needs writable temporary storage:
volumes:
- name: tmp
emptyDir: {}
Then mount it where required.
13. seccompProfile
------------------
Seccomp restricts which Linux system calls a process can make.
Example:
securityContext:
seccompProfile:
type: RuntimeDefault
Common types:
- RuntimeDefault
- Unconfined
- Localhost
RuntimeDefault:
Use container runtime's default seccomp profile.
Unconfined:
No seccomp restrictions.
Localhost:
Use a custom profile already present on the node.
14. seLinuxOptions
------------------
Configures SELinux-related security settings.
Example:
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
More advanced and depends on SELinux configuration.
15. MAJOR SECURITYCONTEXT FIELDS
--------------------------------
For CKA, remember these core fields:
Pod-level:
- runAsUser
- runAsGroup
- runAsNonRoot
- fsGroup
- fsGroupChangePolicy
- supplementalGroups
- seccompProfile
- seLinuxOptions
Container-level:
- runAsUser
- runAsGroup
- runAsNonRoot
- privileged
- allowPrivilegeEscalation
- readOnlyRootFilesystem
- capabilities
- seccompProfile
- seLinuxOptions
16. SIX CORE FIELDS FOR QUICK CKA REVISION
------------------------------------------
1. runAsUser
2. runAsGroup
3. runAsNonRoot
4. fsGroup
5. capabilities
6. allowPrivilegeEscalation
Note:
"Major 6 fields" is a useful study shortcut, NOT an official Kubernetes limit.
17. SECURE POD EXAMPLE
----------------------
apiVersion: v1
kind: Pod
metadata:
name: secure-pod
spec:
securityContext:
runAsUser: 1000
runAsGroup: 3000
runAsNonRoot: true
fsGroup: 2000
containers:
- name: app
image: nginx
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
Meaning:
Pod:
- UID = 1000
- GID = 3000
- Must not run as root
- Volume group = 2000
Container:
- Cannot escalate privileges
- Root filesystem is read-only
- All Linux capabilities are dropped
18. SECURITYCONTEXT vs POD SECURITY ADMISSION
---------------------------------------------
SecurityContext:
Defines HOW a Pod/container runs.
Examples:
- runAsUser
- privileged
- capabilities
- fsGroup
Pod Security Admission:
Defines WHAT security configuration is allowed in a namespace.
Levels:
- Privileged
- Baseline
- Restricted
Think:
Pod Security Admission
↓
"Are you ALLOWED to create this Pod?"
SecurityContext
↓
"HOW will this Pod/container run?"
19. SECURITYCONTEXT vs RBAC
---------------------------
RBAC:
Controls what a user/service account can do to Kubernetes API resources.
Examples:
- get Pods
- delete Deployments
- create Secrets
SecurityContext:
Controls how the container process runs on Linux.
Examples:
- UID
- GID
- root/non-root
- capabilities
- privileges
- filesystem
20. CKA QUICK MEMORY
--------------------
Question:
"Run container as user 1000"
→ runAsUser: 1000
"Run as non-root"
→ runAsNonRoot: true
"Set group ownership for volumes"
→ fsGroup: 2000
"Set primary group"
→ runAsGroup: 3000
"Add/remove Linux privileges"
→ capabilities
"Remove all capabilities"
→ capabilities:
drop:
- ALL
"Don't allow gaining additional privileges"
→ allowPrivilegeEscalation: false
"Make container filesystem read-only"
→ readOnlyRootFilesystem: true
"Run with elevated privileges"
→ privileged: true
"Restrict Linux system calls"
→ seccompProfile
FINAL MENTAL MODEL
------------------
Kubernetes Security
|
+----------------+----------------+
| | |
RBAC Pod Security SecurityContext
Admission |
| | |
API permissions Pod restrictions Linux process
|
+-----------------------------+----------------+
| | | | | |
runAsUser runAsGroup fsGroup capabilities privileged
| | | | |
UID GID Volume Privileges Root
|
allowPrivilegeEscalation
MOST IMPORTANT FOR CKA:
-----------------------
runAsUser
runAsGroup
runAsNonRoot
fsGroup
capabilities
allowPrivilegeEscalation
privileged
readOnlyRootFilesystem
seccompProfile
========================================
KUBERNETES NETWORKPOLICY — CKA REVISION
========================================
1. WHAT IS NETWORKPOLICY?
--------------------------
NetworkPolicy controls network traffic to/from Pods.
It is an ALLOW-LIST model:
- No explicit deny: field
- If no policy isolates a direction -> traffic is allowed by default
- Once a Pod is isolated for a direction -> only explicitly allowed traffic works
- NetworkPolicy is enforced by the CNI/networking plugin
- Our lab uses Calico
Important:
NetworkPolicy is namespace-scoped.
2. BASIC STRUCTURE
------------------
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: my-policy
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Ingress
- Egress
ingress:
- ...
egress:
- ...
3. TOP-LEVEL podSelector
------------------------
spec:
podSelector:
matchLabels:
app: backend
Means:
"Which Pods does THIS policy protect?"
It selects Pods in the SAME namespace as the NetworkPolicy.
podSelector: {}
Means:
"ALL Pods in this namespace."
4. INGRESS vs EGRESS
--------------------
INGRESS:
Traffic coming INTO the selected/protected Pod.
Mental model:
INGRESS = FROM whom can traffic come INTO ME?
Example:
podSelector:
matchLabels:
app: backend
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 80
Meaning:
Backend Pods allow incoming TCP/80 traffic FROM frontend Pods.
Diagram:
Frontend
|
| TCP 80
↓
Backend
EGRESS:
Traffic going OUT FROM the selected/protected Pod.
Mental model:
EGRESS = TO where can traffic go FROM ME?
Example:
podSelector:
matchLabels:
app: frontend
egress:
- to:
- podSelector:
matchLabels:
app: backend
ports:
- protocol: TCP
port: 80
Meaning:
Frontend Pods can send TCP/80 traffic TO backend Pods.
Diagram:
Frontend
|
| TCP 80
↓
Backend
5. EASY MEMORY
--------------
INGRESS:
MY POD <- FROM <- SOURCE
EGRESS:
MY POD -> TO -> DESTINATION
For traffic:
A -> B : 80
Think:
A's Egress
+
B's Ingress
6. from
-------
Used under ingress.
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
Means:
"Allow traffic FROM frontend Pods."
The from selector identifies the SOURCE.
7. to
------
Used under egress.
egress:
- to:
- podSelector:
matchLabels:
app: backend
Means:
"Allow traffic TO backend Pods."
The to selector identifies the DESTINATION.
8. podSelector IN PEER
----------------------
Example:
egress:
- to:
- podSelector:
matchLabels:
app: backend
Means destination Pods have app=backend.
Important:
Top-level podSelector and peer podSelector have different purposes.
Top-level:
"Which Pods am I protecting?"
Peer podSelector:
"Which Pods can communicate with those protected Pods?"
9. namespaceSelector
--------------------
Selects namespaces using namespace labels.
Example:
namespaceSelector:
matchLabels:
team: app
Means:
Select namespaces having:
team=app
10. namespaceSelector + podSelector
-----------------------------------
When BOTH are in the SAME peer item:
- namespaceSelector:
matchLabels:
team: app
podSelector:
matchLabels:
app: backend
This means:
Namespace MUST match
AND
Pod MUST match
So:
namespace team=app
AND
pod app=backend
11. SEPARATE PEER ITEMS = OR
----------------------------
Example:
from:
- podSelector:
matchLabels:
app: frontend
- namespaceSelector:
matchLabels:
team: app
Means:
frontend Pod
OR
Pod from namespace team=app
MEMORY:
Same peer item = AND
Separate list items = OR
12. ipBlock
-----------
Used to select IP/CIDR ranges.
Example:
from:
- ipBlock:
cidr: 192.168.10.0/24
Means:
Allow traffic from that IP range.
except excludes a subrange:
ipBlock:
cidr: 10.10.0.0/16
except:
- 10.10.5.0/24
Important:
ipBlock is IP-based selection.
Do not think of it simply as "external traffic only".
Actual behavior can depend on networking implementation.
13. PORTS
---------
ports controls destination port/protocol.
Example:
ports:
- protocol: TCP
port: 80
Means:
Allow TCP traffic to destination port 80.
Common protocols:
TCP
UDP
SCTP
14. endPort
----------
Allows a port range.
Example:
ports:
- protocol: TCP
port: 8000
endPort: 8080
Means:
TCP ports 8000 through 8080.
15. DEFAULT DENY — INGRESS
--------------------------
podSelector: {}
policyTypes:
- Ingress
No ingress rules.
Meaning:
All Pods in this namespace become ingress-isolated
and no ingress traffic is allowed by this policy.
Mental model:
ALL Pods
↓
Ingress
↓
DENY unless another applicable policy allows it
16. DEFAULT DENY — EGRESS
-------------------------
podSelector: {}
policyTypes:
- Egress
No egress rules.
Meaning:
All Pods in this namespace become egress-isolated
and no egress traffic is allowed by this policy.
This also blocks DNS unless DNS egress is explicitly allowed.
17. DEFAULT DENY — BOTH
-----------------------
podSelector: {}
policyTypes:
- Ingress
- Egress
Meaning:
All Pods in namespace are isolated for both directions.
Ingress -> blocked unless allowed
Egress -> blocked unless allowed
18. DEFAULT DENY + ALLOW
------------------------
Common real-world/CKA pattern:
1. Apply default deny
2. Add specific allow policy
Example:
Default:
ALL traffic denied
Then:
frontend -> backend TCP 80 allowed
Result:
frontend -> backend :80 ALLOWED
frontend -> database :3306 BLOCKED
frontend -> other Pod BLOCKED
19. MULTIPLE NETWORKPOLICIES
----------------------------
NetworkPolicies are ADDITIVE.
They do NOT override each other.
Example:
Policy A:
allows frontend
Policy B:
allows monitoring
Result:
frontend OR monitoring can communicate.
Think:
Policy A allowed traffic
+
Policy B allowed traffic
=
UNION of allowed traffic
A later policy does NOT automatically override an earlier policy.
20. POLICY TYPES
----------------
policyTypes tells Kubernetes which directions the policy isolates.
Ingress:
Controls incoming traffic.
Egress:
Controls outgoing traffic.
Both:
Controls both directions.
21. IMPORTANT DIRECTION RULE
----------------------------
For:
Frontend -> Backend
Frontend:
Egress
Backend:
Ingress
If frontend is egress-isolated:
Frontend's Egress must allow it.
If backend is ingress-isolated:
Backend's Ingress must allow it.
If both are isolated:
BOTH sides need applicable allow rules.
22. DNS + EGRESS
----------------
DNS is also network traffic.
If a Pod is egress-isolated:
DNS will be blocked unless explicitly allowed.
Typical DNS:
UDP 53
TCP 53
Example:
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
Meaning:
Frontend
|
| DNS UDP/TCP 53
↓
CoreDNS
23. DNS TROUBLESHOOTING
----------------------
Check DNS Service:
kubectl get svc -n kube-system
Check namespace labels:
kubectl get ns kube-system --show-labels
Check CoreDNS labels:
kubectl get pods -n kube-system --show-labels
Check Pod DNS configuration:
kubectl exec <pod> -- cat /etc/resolv.conf
Typical:
nameserver 10.96.0.10
Then test:
kubectl exec <pod> -- nslookup kubernetes.default
IMPORTANT:
Do not blindly assume namespace or CoreDNS labels.
Inspect the actual cluster labels.
24. NETWORKPOLICY + SERVICE
---------------------------
Service provides stable networking/routing to Pods.
Example:
frontend
|
↓
backend-service
|
↓
backend Pods
Service:
- provides stable name/IP
- selects backend Pods
- routes traffic to endpoints
NetworkPolicy:
- controls allowed Pod traffic
- does NOT have a "serviceName" selector
There is no:
serviceName: backend-service
inside NetworkPolicy peers.
Think:
Service:
"Where should traffic be routed?"
NetworkPolicy:
"Is this Pod traffic allowed?"
25. SERVICE SELECTOR + ENDPOINTS
-------------------------------
Example Service:
selector:
app: backend
It finds Pods with:
app=backend
Those Pods become Service endpoints.
Check:
kubectl get svc <service-name>
kubectl get endpoints <service-name>
kubectl get endpointslice
If Service has no endpoints:
NetworkPolicy can be correct,
but Service traffic still fails because there is
no backend Pod to receive it.
26. SERVICE PORT vs targetPort
------------------------------
Example:
ports:
- port: 80
targetPort: 8080
Client accesses:
Service :80
Service forwards to:
Pod :8080
Conceptually:
Client
|
| :80
↓
Service
|
| → :8080
↓
Pod
For NetworkPolicy troubleshooting,
remember that policy concerns actual Pod traffic,
and exact Service translation/enforcement details
can depend on the networking implementation.
Do not blindly memorize:
"NetworkPolicy always uses Service port"
or
"NetworkPolicy always uses targetPort."
27. NETWORKPOLICY + SERVICE IMPORTANT POINT
-------------------------------------------
NetworkPolicy does not select a Service by name.
It selects traffic using:
podSelector
namespaceSelector
ipBlock
and controls ports using:
ports
protocol
endPort
28. CNI
-------
Kubernetes defines the NetworkPolicy API.
The networking/CNI plugin actually enforces it.
Mental model:
NetworkPolicy
↓
CNI / networking plugin
↓
Actual network enforcement
Your cluster:
Calico
We identified:
calico-node
calico-kube-controllers
So Calico is responsible for enforcing
NetworkPolicy in your lab.
29. CNI LIMITATION
------------------
NetworkPolicy behavior depends on CNI/networking implementation.
Not every networking implementation supports
every advanced networking feature identically.
For CKA:
Understand the concept:
Kubernetes API defines NetworkPolicy,
CNI implements/enforces it.
30. NETWORKPOLICY TROUBLESHOOTING
---------------------------------
If traffic is failing, check in this order:
1. Does NetworkPolicy exist?
kubectl get networkpolicy -n <namespace>
2. Inspect it:
kubectl describe networkpolicy <name> -n <namespace>
3. Check Pod labels:
kubectl get pods --show-labels -n <namespace>
4. Check namespace labels:
kubectl get ns --show-labels
5. Verify podSelector matches the intended Pods.
6. Verify from/to selector.
7. Verify namespaceSelector.
8. Verify port/protocol.
9. If DNS fails, check DNS policy.
10. Check Service:
kubectl get svc
11. Check endpoints:
kubectl get endpoints
kubectl get endpointslice
12. Check whether application is listening
on the expected port.
13. Check CNI/networking plugin.
31. COMMON CKA COMMANDS
-----------------------
kubectl get networkpolicy -n <namespace>
kubectl describe networkpolicy <name> -n <namespace>
kubectl get pods --show-labels -n <namespace>
kubectl get ns --show-labels
kubectl get svc -n <namespace>
kubectl get endpoints -n <namespace>
kubectl get endpointslice -n <namespace>
kubectl exec <pod> -- nc -vz <IP> <port>
kubectl exec <pod> -- nslookup <service>
kubectl exec <pod> -- cat /etc/resolv.conf
32. MOST IMPORTANT CKA MENTAL MODEL
-----------------------------------
Top-level podSelector
↓
"WHO AM I PROTECTING?"
Ingress
↓
"WHO CAN COME TO ME?"
↓
from
Egress
↓
"WHERE CAN I GO?"
↓
to
33. FINAL MEMORY TRICK
----------------------
INGRESS:
MY POD
↑
FROM
↑
SOURCE
"Who can come INTO me?"
EGRESS:
MY POD
↓
TO
↓
DESTINATION
"Where can I send traffic?"
34. EXAMPLE — COMPLETE
----------------------
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: frontend-policy
spec:
podSelector:
matchLabels:
app: frontend
policyTypes:
- Egress
egress:
# Frontend → Backend
- to:
- namespaceSelector:
matchLabels:
team: app
podSelector:
matchLabels:
app: backend
ports:
- protocol: TCP
port: 80
# Frontend → CoreDNS
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
Interpretation:
Top podSelector:
frontend = protected/source Pod
First egress rule:
frontend → backend :80 = ALLOWED
Second egress rule:
frontend → CoreDNS :53 = ALLOWED
Anything else from frontend:
BLOCKED because frontend is egress-isolated.
========================================
NETWORKPOLICY — 10 SECOND REVISION
========================================
podSelector
= Which Pods am I protecting?
Ingress
= Who can come INTO me?
from
= SOURCE
Egress
= Where can I send traffic?
to
= DESTINATION
namespaceSelector
= Select namespaces by labels
podSelector
= Select Pods by labels
ipBlock
= Select IP/CIDR
ports
= Destination port/protocol
Same peer item:
namespaceSelector + podSelector = AND
Separate peer items = OR
podSelector: {}
= All Pods in that namespace
policyTypes:
Ingress = isolate incoming traffic
Egress = isolate outgoing traffic
No applicable isolation:
traffic normally allowed
Isolated direction:
only explicitly allowed traffic works
Multiple policies:
ADDITIVE / UNION
Egress isolation:
remember DNS!
Service:
routes/selects Pods
NetworkPolicy:
controls Pod traffic
CNI:
actually enforces NetworkPolicy
========================================
## 🧱 1. The Core Infrastructure Components
Before any traffic moves, Calico and Kubernetes set up the landscape:
* The veth Pairs: For every Pod created, Calico creates a virtual ethernet cable. One end sits in the Pod's network namespace; the other end sits in the host namespace (named caliXXXXX).
* The Map (Routing Table): Calico’s agent (Felix) pre-populates the Linux kernel’s native routing table (ip route). It writes rules mapping every Pod IP to its specific caliXXXXX interface.
* The Security Guard (Netfilter/iptables): Calico and kube-proxy inject custom sub-chains (starting with cali- and KUBE-) directly into the kernel's native Netfilter hooks.
------------------------------
## 🕒 2. The Strict Order of Operations
When a packet travels from Pod 1 to a Service VIP, it triggers Netfilter hooks and routing decisions in a strict, unchangeable timeline:
## Hook 1: PREROUTING (The Entry Checkpoint)
* kube-proxy hits first: It intercepts the virtual Service VIP and executes DNAT, rewriting the destination address to a Real Backend Pod IP.
* Calico hits immediately after: It runs its cali-PREROUTING chain. It inspects the packet after DNAT and applies Ingress Network Policies.
* Action: It either permits the packet or drops it instantly to save CPU cycles.
## Step 2: The Routing Decision (The Intersection)
* The Linux Kernel takes over: The kernel looks at the packet's destination (the Real Pod IP).
* The Map Lookup: It references the routing table that Calico pre-configured.
* Action: The kernel matches the Pod IP and determines the exact exit interface (caliXXXXX for same-host, or a physical NIC like eth0 for cross-host).
## Hook 3: FORWARD (The Transit Lane)
* The Context: Because the packet is crossing from one network interface namespace to another, the kernel routes it through the FORWARD hook.
* Calico hits here: It runs its cali-FORWARD chain to validate transit safety.
* Action: It double-checks that Pod 1 is allowed to talk to Pod 2. If a policy blocks it, the packet is killed here.
## Hook 4: POSTROUTING (The Exit Gate)
* The Context: The packet has been routed and cleared by security. It is sitting at the exit interface, ready to leave.
* Calico/kube-proxy hit here: They run the cali-POSTROUTING chain to enforce Egress Network Policies.
* Action: If the packet is exiting the cluster to the external internet, SNAT (Masquerading) is applied at the very last second, rewriting the source IP to the Node's IP so replies can find their way back.
------------------------------
## 🎯 3. Visual Execution Flow
[ Packet Leaves Pod 1 via veth ] ──► Enters Host Namespace
│
▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│ KERNEL HOOKS & CALICO INTERACTION │
│ │
│ 1. PREROUTING HOOK │
│ ├──► kube-proxy: DNAT (VIP ──► Real Pod 2 IP) │
│ └──► Calico: Checks Ingress Policy ──► [ALLOW / DROP] │
│ │
│ 2. ROUTING DECISION │
│ └──► Linux Kernel: Reads Calico's pre-written routing table │
│ └──► Destination matched ──► Target Interface locked in │
│ │
│ 3. FORWARD HOOK │
│ └──► Calico: Checks Transit/Inter-pod Policy ──► [ALLOW / DROP] │
│ │
│ 4. POSTROUTING HOOK │
│ ├──► Calico: Checks Egress Policy ──► [ALLOW / DROP] │
│ └──► kube-proxy/Calico: Applies SNAT (only if traffic leaves node/cluster) │
└─────────────────────────────────────────────────────────────────────────────────1
│
▼
[ Packet Enters Target veth ] ──► Arrives safely at Pod 2!
------------------------------
## 💡 4. The Golden Rules to Remember
1. Linux Kernel does the driving: Calico never routes a packet mid-flight. The kernel does the routing based on a map Calico drew ahead of time.
2. Calico is the firewall: Calico's primary job inside the Netfilter hooks is executing immediate Allow/Deny decisions based on your Network Policies.
3. DNAT changes the target, Routing finds the path, SNAT changes the source: DNAT happens first so the kernel knows where to route the packet. SNAT happens last so it doesn't mess up the routing decision or Calico's policy evaluation. [1]
Now that you have the complete summary, where would you like to go next?
* We can look at how this changes if you switch Calico to VXLAN/IPIP Overlay mode instead of native routing.
* We can look at how Calico's eBPF mode deletes all of these iptables hooks entirely for faster networking.
POD MASTERY
│
├── 1. Pod Fundamentals
│ ├── Pod Concept
│ ├── Smallest Deployable Unit
│ ├── Pod vs Container
│ ├── Multi-Container Pod
│ ├── Shared Network Namespace
│ ├── Shared Storage
│ └── Shared Pod Resources
│
├── 2. Pod Lifecycle
│ ├── Pod Phases
│ │ ├── Pending
│ │ ├── Running
│ │ ├── Succeeded
│ │ ├── Failed
│ │ └── Unknown
│ │
│ ├── Container States
│ │ ├── Waiting
│ │ ├── Running
│ │ └── Terminated
│ │
│ ├── Restart Policy
│ │ ├── Always
│ │ ├── OnFailure
│ │ └── Never
│ │
│ ├── Kubelet Container Monitoring
│ └── Container Exit Status
│
├── 3. Pod Initialization & Containers
│ ├── Init Containers
│ ├── Init Container Ordering
│ ├── Init Container Success/Failure
│ ├── Multiple Init Containers
│ ├── Multi-Container Pods
│ ├── Container Communication
│ └── Sidecar Concept
│
├── 4. Pod Networking
│ ├── Pod Network Namespace
│ ├── Pod IP
│ ├── Container-to-Container Communication
│ ├── localhost Inside Pod
│ ├── Network Namespace Sharing
│ ├── veth Pair
│ ├── CNI
│ └── Pod Network Flow
│
├── 5. Pod Storage
│ ├── Volume Concept
│ ├── Volume Lifecycle
│ │
│ ├── emptyDir
│ │ ├── Pod Lifetime
│ │ ├── Container Restart
│ │ └── Shared Between Containers
│ │
│ ├── hostPath
│ │ ├── Node Filesystem
│ │ ├── Node Dependency
│ │ └── DirectoryOrCreate
│ │
│ ├── ConfigMap Volume
│ ├── Secret Volume
│ │ ├── data
│ │ ├── stringData
│ │ └── Base64 Encoding
│ │
│ ├── Projected Volume
│ │ ├── ConfigMap
│ │ ├── Secret
│ │ ├── Downward API
│ │ └── ServiceAccount Token
│ │
│ └── Downward API
│ ├── Pod Metadata
│ ├── Pod Name
│ ├── Namespace
│ └── Pod Fields
│
├── 6. Pod Resources
│ ├── Resource Requests
│ ├── Resource Limits
│ ├── CPU
│ ├── Memory
│ ├── Ephemeral Storage
│ ├── Scheduler Uses Requests
│ ├── Limits Enforcement
│ ├── QoS Classes
│ │ ├── Guaranteed
│ │ ├── Burstable
│ │ └── BestEffort
│ ├── OOMKilled
│ └── Resource Pressure
│
├── 7. Pod Health & Probes
│ ├── Why Probes Are Needed
│ ├── Liveness Probe
│ ├── Readiness Probe
│ ├── Startup Probe
│ ├── Probe Types
│ │ ├── HTTP
│ │ ├── TCP
│ │ ├── Exec
│ │ └── GRPC
│ ├── Probe Timing
│ │ ├── initialDelaySeconds
│ │ ├── periodSeconds
│ │ ├── timeoutSeconds
│ │ ├── failureThreshold
│ │ └── successThreshold
│ └── Probe Failure Behavior
│
├── 8. Pod Identity & RBAC
│ ├── ServiceAccount
│ ├── Default ServiceAccount
│ ├── serviceAccountName
│ ├── ServiceAccount Token
│ ├── automountServiceAccountToken
│ ├── Authentication
│ ├── Authorization
│ ├── Role
│ ├── ClusterRole
│ ├── RoleBinding
│ ├── ClusterRoleBinding
│ ├── Namespace Scope
│ └── Cluster-Wide Scope
│
├── 9. Pod SecurityContext
│ ├── Why SecurityContext
│ ├── Pod-Level vs Container-Level
│ ├── runAsUser
│ ├── runAsGroup
│ ├── supplementalGroups
│ ├── fsGroup
│ ├── runAsNonRoot
│ ├── readOnlyRootFilesystem
│ ├── Linux Capabilities
│ │ ├── add
│ │ └── drop
│ ├── privileged
│ ├── allowPrivilegeEscalation
│ └── seccompProfile
│ ├── RuntimeDefault
│ ├── Unconfined
│ └── Localhost
│
├── 10. Pod Security Standards
│ ├── PSS
│ │ ├── Privileged
│ │ ├── Baseline
│ │ └── Restricted
│ │
│ ├── Pod Security Admission
│ ├── Admission Phase
│ ├── Namespace Labels
│ ├── enforce
│ ├── warn
│ └── audit
│
├── 11. Pod Scheduling
│ ├── Scheduler Flow
│ ├── nodeName
│ ├── nodeSelector
│ ├── Node Affinity
│ ├── Pod Affinity
│ ├── Pod Anti-Affinity
│ ├── topologyKey
│ ├── Taints
│ │ ├── NoSchedule
│ │ ├── PreferNoSchedule
│ │ └── NoExecute
│ └── Tolerations
│ ├── Equal
│ └── Exists
│
├── 12. Pod Termination
│ ├── Pod Deletion
│ ├── Termination Grace Period
│ ├── preStop Hook
│ ├── SIGTERM
│ ├── SIGKILL
│ └── Graceful Shutdown
│
├── 13. Pod State & Conditions
│ ├── Pod Phase
│ ├── Pod Conditions
│ │ ├── PodScheduled
│ │ ├── Initialized
│ │ ├── ContainersReady
│ │ └── Ready
│ ├── Container State
│ └── Phase vs Condition vs Container State
│
├── 14. Pod DNS
│ ├── Pod Hostname
│ ├── /etc/hosts
│ ├── /etc/resolv.conf
│ ├── CoreDNS
│ ├── DNS Search Domains
│ ├── ndots
│ ├── dnsPolicy
│ └── hostNetwork Interaction
│
├── 15. Pod Eviction & Node Pressure
│ ├── MemoryPressure
│ ├── DiskPressure
│ ├── PIDPressure
│ ├── Ephemeral Storage Pressure
│ ├── Kubelet Eviction
│ ├── QoS and Eviction
│ └── OOMKilled vs Eviction
│
├── 16. Static Pods
│ ├── Static Pod Concept
│ ├── Kubelet Managed
│ ├── Static Pod Manifest
│ ├── /etc/kubernetes/manifests/
│ ├── Static Pod vs Normal Pod
│ └── Control Plane Static Pods
│
└── 17. Pod Troubleshooting & Final Mastery
├── Pending
├── CrashLoopBackOff
├── ImagePullBackOff
├── CreateContainerConfigError
├── ContainerCreating
├── OOMKilled
├── Permission Denied
├── Volume Mount Failure
├── ConfigMap/Secret Failure
├── Probe Failure
├── SecurityContext Failure
├── DNS Failure
├── Scheduling Failure
├── Node Pressure / Eviction
└── Final Pod Mastery Lab & Test
==========================================================
MODULE 4 - HEALTH PROBES (REVISION NOTES)
==========================================================
##########################################################
1. WHY DO WE NEED PROBES?
##########################################################
A container can be:
1. Starting
2. Running but not ready for traffic
3. Running and healthy
4. Running but hung/dead internally
Kubelet cannot determine all of these states by only checking if
the process exists.
Therefore Kubernetes provides three probes:
1. Startup Probe
2. Readiness Probe
3. Liveness Probe
----------------------------------------------------------
##########################################################
2. STARTUP PROBE
##########################################################
Purpose:
--------
Checks whether an application can successfully complete startup.
Question it answers:
--------------------
"Can this application finish starting?"
Used for:
---------
- Slow starting applications
- Spring Boot
- Java applications
- Applications loading cache
- Database initialization
How it works:
-------------
Container Starts
↓
Startup Probe runs
↓
If Fail
↓
Keep checking
↓
If Success (ONLY ONCE)
↓
Startup Probe is DISABLED forever
↓
Readiness and Liveness begin
Important:
----------
• Runs only during startup.
• Runs until first success.
• After success it never runs again for that container.
• If container restarts, Startup Probe starts again because it is a NEW container instance.
Configuration:
--------------
initialDelaySeconds
periodSeconds
failureThreshold
timeoutSeconds
Maximum startup time allowed:
initialDelaySeconds +
(periodSeconds × failureThreshold)
If startup never succeeds within this time:
→ Kubelet restarts container.
----------------------------------------------------------
##########################################################
3. READINESS PROBE
##########################################################
Purpose:
--------
Determines whether a Pod should receive traffic.
Question it answers:
--------------------
"Can I serve user requests?"
If Readiness succeeds:
Ready=True
↓
EndpointSlice Controller adds Pod IP
↓
kube-proxy updates iptables/IPVS
↓
Service sends traffic
If Readiness fails:
Ready=False
↓
EndpointSlice Controller removes Pod IP
↓
kube-proxy updates iptables/IPVS
↓
Service stops sending traffic
IMPORTANT:
----------
Readiness NEVER restarts a container.
Container continues running.
Only traffic is stopped.
----------------------------------------------------------
##########################################################
4. LIVENESS PROBE
##########################################################
Purpose:
--------
Determines whether the application is still healthy.
Question it answers:
--------------------
"Am I still alive?"
If Liveness succeeds:
Nothing happens.
Application keeps running.
If Liveness fails:
FailureThreshold reached
↓
Kubelet kills container
↓
RestartPolicy checked
↓
Container restarted
Repeated failures:
Restart
↓
Restart
↓
Restart
↓
Exponential Backoff
↓
CrashLoopBackOff
IMPORTANT:
----------
Liveness DOES restart containers.
----------------------------------------------------------
##########################################################
5. EXECUTION ORDER
##########################################################
Container Starts
│
▼
Startup Probe
│
▼
Startup Success
│
├─────────────┐
▼ ▼
Readiness Liveness
│ │
▼ ▼
Traffic Restart if unhealthy
----------------------------------------------------------
##########################################################
6. KUBELET RESPONSIBILITIES
##########################################################
Kubelet:
• Runs Startup Probe
• Runs Readiness Probe
• Runs Liveness Probe
• Updates Pod Ready condition
• Restarts failed containers
• Applies Restart Policy
----------------------------------------------------------
##########################################################
7. STARTUP vs READINESS vs LIVENESS
##########################################################
Startup Probe
Purpose:
Can application finish startup?
Controls:
Startup phase only
Restarts:
YES (if startup never succeeds)
Traffic:
NO
----------------------------------------------------------
Readiness Probe
Purpose:
Can application receive traffic?
Controls:
Traffic routing
Restarts:
NO
Traffic:
YES
----------------------------------------------------------
Liveness Probe
Purpose:
Is application still healthy?
Controls:
Container restart
Restarts:
YES
Traffic:
NO
----------------------------------------------------------
##########################################################
8. STARTUP PROBE vs LIVENESS initialDelaySeconds
##########################################################
Liveness initialDelaySeconds
Meaning:
"Wait X seconds.
After that I expect application to be healthy."
Problem:
If startup takes longer than expected,
container gets killed repeatedly.
----------------------------------------------------------
Startup Probe
Meaning:
"I don't know how long startup takes.
I'll keep checking until startup succeeds
or maximum startup time expires."
This removes the need to guess startup time.
----------------------------------------------------------
##########################################################
9. CRASHLOOPBACKOFF
##########################################################
CrashLoopBackOff is NOT a Pod phase.
It is kubectl STATUS shown when:
Container repeatedly exits
AND
RestartPolicy causes restart
AND
Kubelet applies exponential restart delay.
Typical delays:
Restart immediately
↓
10 sec
↓
20 sec
↓
40 sec
↓
80 sec
↓
160 sec
↓
300 sec (maximum backoff)
After maximum delay,
Kubelet still continues retrying approximately every 300 seconds
until the Pod is deleted or the problem is fixed.
----------------------------------------------------------
##########################################################
10. PROBE TIMELINE
##########################################################
Container Starts
│
▼
Startup Probe
│
├── Fail → Keep waiting
│
└── Success
│
▼
Startup Probe Disabled
│
▼
Readiness Starts
│
▼
Pod Ready=True
│
▼
Service sends traffic
│
▼
Application hangs
│
├───────────────┐
▼ ▼
Readiness Fail Liveness Fail
│ │
▼ ▼
Stop Traffic Restart Container
----------------------------------------------------------
##########################################################
11. IMPORTANT CKA POINTS
##########################################################
✓ Startup runs only until first success.
✓ Startup runs again after every container restart.
✓ Readiness controls traffic only.
✓ Readiness NEVER restarts containers.
✓ Liveness restarts unhealthy containers.
✓ Startup disables Readiness and Liveness until startup succeeds.
✓ EndpointSlice Controller reacts to Ready=True/False.
✓ kube-proxy updates iptables/IPVS after EndpointSlice changes.
✓ Services only send traffic to Ready Pods.
✓ CrashLoopBackOff is caused by repeated restarts with exponential backoff.
==========================================================
END OF MODULE 4
==========================================================
===========================================================
CKA POD MASTERY
MODULE 4 – HEALTH PROBES (MASTER NOTES)
===========================================================
############################################################
WHY KUBERNETES NEEDS PROBES
############################################################
Kubelet can only see:
Container Process
Running?
Exit Code?
It CANNOT know:
✓ Is application still loading?
✓ Is application ready?
✓ Is application hung?
✓ Can application serve requests?
Therefore Kubernetes provides three probes.
------------------------------------------------------------
STARTUP PROBE
Question:
"Can application finish startup?"
READINESS PROBE
Question:
"Can application receive traffic?"
LIVENESS PROBE
Question:
"Is application still healthy?"
------------------------------------------------------------
Think like this:
Startup
↓
Readiness
↓
Liveness
============================================================
STARTUP PROBE
============================================================
Purpose
Protect slow-starting applications.
Examples
Spring Boot
Java Applications
Large Cache Loading
Database Migration
Machine Learning Models
Flow
Container Starts
↓
Startup Probe Runs
↓
Fail
↓
Keep Waiting
↓
Fail
↓
Keep Waiting
↓
Success
↓
Disable Startup Probe Forever
↓
Enable Readiness
↓
Enable Liveness
IMPORTANT
Startup Probe runs ONLY during startup.
Once it succeeds once,
it NEVER runs again
for THAT container.
If container restarts,
Startup Probe starts again.
WHY?
Because probes belong to
container instance.
Container #1
↓
Startup Passed
↓
Container Dies
↓
Container #2
↓
Startup runs AGAIN
============================================================
REAL EXAMPLE
============================================================
Application Startup
sleep 30
touch /tmp/startup-ok
touch /tmp/ready
sleep 300
Startup checks
cat /tmp/startup-ok
Timeline
0 sec
Container Starts
↓
Startup Probe
↓
Fail
↓
Fail
↓
touch /tmp/startup-ok
↓
Startup Success
↓
Startup Disabled
↓
Readiness Starts
↓
Liveness Starts
============================================================
READINESS PROBE
============================================================
Purpose
Should this Pod receive traffic?
If Ready=True
↓
API Server updated
↓
EndpointSlice Controller
↓
Pod IP Added
↓
kube-proxy
↓
iptables/IPVS updated
↓
Traffic Starts
-------------------------------------
If Ready=False
↓
API Server updated
↓
EndpointSlice Controller
↓
Remove Pod IP
↓
kube-proxy updates
↓
NO TRAFFIC
IMPORTANT
Readiness NEVER
kills
restarts
or recreates container.
It ONLY controls traffic.
============================================================
REAL EXAMPLE
============================================================
Application Running
↓
rm /tmp/ready
↓
Readiness Probe
↓
Fail
↓
Ready=False
↓
EndpointSlice removes Pod
↓
Service Stops Sending Traffic
Container STILL RUNNING
============================================================
LIVENESS PROBE
============================================================
Purpose
Is application still alive?
If Healthy
↓
Nothing
If Unhealthy
↓
Failure Threshold Reached
↓
Kubelet
↓
Kill Container
↓
Restart Policy
↓
Restart Container
============================================================
REAL EXAMPLE
============================================================
Application
touch /tmp/ready
↓
Later
rm /tmp/ready
↓
Liveness
Fail
↓
Fail
↓
Fail
↓
Kubelet Restarts Container
============================================================
STARTUP vs READINESS vs LIVENESS
============================================================
Startup
Protect Startup
Runs once
Can Restart
No Traffic Control
-------------------------------------
Readiness
Traffic Control
Runs continuously
Never Restarts
Controls EndpointSlice
-------------------------------------
Liveness
Health Check
Runs continuously
Restarts Container
No Traffic Control
============================================================
INITIAL DELAY
============================================================
Startup initialDelay
Wait before checking startup.
Useful when app
ALWAYS needs
some minimum time.
------------------------------------------------
Liveness initialDelay
Wait before checking health.
Problem
You must GUESS startup time.
If app starts slower
↓
Liveness kills container.
This is WHY Startup Probe exists.
============================================================
FAILURE THRESHOLD
============================================================
Example
periodSeconds = 10
failureThreshold = 3
Three consecutive failures
↓
Action
Readiness
↓
NotReady
Liveness
↓
Restart
Startup
↓
Startup Failed
Restart Container
============================================================
CRASHLOOPBACKOFF
============================================================
CrashLoopBackOff
means
Container keeps restarting.
Kubelet applies
Exponential Delay
Typical
Restart
↓
10 sec
↓
20 sec
↓
40 sec
↓
80 sec
↓
160 sec
↓
300 sec
↓
300 sec
↓
300 sec
continues forever
until fixed.
============================================================
COMPLETE TIMELINE
============================================================
Container Starts
↓
Startup Probe
↓
Success
↓
Readiness
↓
Ready=True
↓
Service sends traffic
↓
Application hangs
↓
Readiness fails
↓
Traffic Stops
↓
Liveness fails
↓
Restart
↓
NEW CONTAINER
↓
Startup Probe AGAIN
============================================================
MY MISTAKES (VERY IMPORTANT)
============================================================
Mistake 1
I wrote
command:
- sleep
- 30
- touch /tmp/file
Wrong
Reason
command is NOT shell script.
First value
Executable
Remaining values
Arguments
Correct
command:
- sh
- -c
- |
sleep 30
touch /tmp/file
------------------------------------------------
Mistake 2
I thought
CrashLoopBackOff
means Pod stopped.
Reality
Pod can show
Running
while kubelet
is retrying container.
CrashLoopBackOff
is restart state,
NOT pod phase.
------------------------------------------------
Mistake 3
I thought
Readiness restart container.
Reality
Readiness NEVER restarts.
Only removes traffic.
------------------------------------------------
Mistake 4
I thought
Startup never runs again.
Reality
Startup runs again
AFTER EVERY CONTAINER RESTART.
------------------------------------------------
Mistake 5
Confused with seconds.
Solution
STOP calculating time.
Instead
Draw Timeline.
Example
Container
↓
Startup
↓
Ready
↓
Traffic
↓
Readiness Fail
↓
Traffic Stops
↓
Liveness Fail
↓
Restart
Much easier than counting seconds.
============================================================
CKA EXAM POINTS
============================================================
✓ Startup protects startup.
✓ Readiness controls traffic.
✓ Liveness restarts containers.
✓ Startup disables
Readiness & Liveness
until startup succeeds.
✓ Readiness modifies
EndpointSlice.
✓ kube-proxy updates
iptables/IPVS.
✓ Services send traffic
ONLY to Ready Pods.
✓ Startup belongs to
container instance.
✓ CrashLoopBackOff
uses exponential backoff.
============================================================
END OF MODULE 4
============================================================
# Kubernetes Pod Mastery - Module 2 Notes
## Process Lifecycle, Restart Policies, Command & Args
------------------------------------------------------------
1. CONTAINER PROCESS LIFECYCLE
------------------------------------------------------------
Every container has ONE main process (PID 1).
The lifecycle of the container depends on this process.
• If the main process is running
→ Container = Running
• If the main process exits with Exit Code 0
→ Container = Completed
• If the main process exits with a non-zero Exit Code
→ Container = Error
Examples:
sleep 300
→ Running
echo "Hello Kubernetes"
→ Completed (Exit Code 0)
false
→ Error (Exit Code 1)
------------------------------------------------------------
2. RESTART POLICIES
------------------------------------------------------------
The kubelet monitors the main process of every container.
When the process exits, kubelet checks the Pod's restartPolicy before deciding whether to create a new container.
There are three restart policies.
A) restartPolicy: Never
• Never restart the container.
Exit Code 0
→ Completed
Exit Code 1
→ Error
------------------------------------------------------------
B) restartPolicy: OnFailure
• Restart only when the process exits with a non-zero Exit Code.
Exit Code 0
→ No Restart
Exit Code 1
→ Restart
------------------------------------------------------------
C) restartPolicy: Always
• Restart the container regardless of whether it exits successfully or fails.
Exit Code 0
→ Restart
Exit Code 1
→ Restart
------------------------------------------------------------
3. CrashLoopBackOff
------------------------------------------------------------
CrashLoopBackOff is NOT a restart policy.
It is a waiting state where kubelet repeatedly restarts a container and applies an exponential backoff delay before trying again.
CrashLoopBackOff can occur when:
• Application exits with Exit Code 1 repeatedly.
OR
• Application exits successfully (Exit Code 0) very quickly (for example sleep 2) while restartPolicy is Always.
The kubelet gradually increases the delay between restart attempts.
------------------------------------------------------------
4. COMMAND AND ARGS
------------------------------------------------------------
Kubernetes uses:
command
→ Executable
args
→ Arguments passed to the executable
Final execution is conceptually:
command[] + args[]
Example:
command:
- echo
- hello
args:
- kubernetes
Final execution:
echo hello kubernetes
The first element of command is the executable.
The remaining elements of command become arguments.
The args array is appended after those arguments.
------------------------------------------------------------
5. Docker ENTRYPOINT and CMD
------------------------------------------------------------
Docker images contain default startup information.
Docker ENTRYPOINT
=
Kubernetes command
Docker CMD
=
Kubernetes args
Images also store metadata like:
• ENTRYPOINT
• CMD
• ENV
• USER
• WORKDIR
• EXPOSE
• LABEL
etc.
------------------------------------------------------------
6. OVERRIDE RULES
------------------------------------------------------------
Case 1
No command
No args
Result:
Use Image ENTRYPOINT
+
Use Image CMD
------------------------------------------------------------
Case 2
Only args
Result:
Use Image ENTRYPOINT
Replace Image CMD with Pod args
------------------------------------------------------------
Case 3
Only command
Result:
Use Pod command
Ignore Image ENTRYPOINT
Ignore Image CMD
------------------------------------------------------------
Case 4
command + args
Result:
Use Pod command
Use Pod args
Ignore Image ENTRYPOINT
Ignore Image CMD
------------------------------------------------------------
7. HOW KUBERNETES STARTS A CONTAINER
------------------------------------------------------------
kubectl apply
|
V
API Server
|
V
Scheduler
|
V
Kubelet
|
V
Container Runtime (containerd / CRI-O)
|
V
OCI Runtime (runc)
|
V
Linux execve()
|
V
Container Starts
------------------------------------------------------------
8. INTERNAL FLOW
------------------------------------------------------------
The image is NEVER modified.
The image already contains metadata:
• ENTRYPOINT
• CMD
• ENV
• USER
• WORKDIR
Containerd pulls the image and reads this metadata.
Kubelet reads the Pod specification.
Kubelet sends any overrides such as:
• command
• args
• environment variables
• volume mounts
• security settings
Containerd combines:
Image defaults
+
Pod overrides
Containerd generates the OCI Runtime Specification.
Finally, runc executes the Linux process using execve().
------------------------------------------------------------
9. IMPORTANT CKA POINTS
------------------------------------------------------------
✓ Every container has one main process.
✓ Exit Code 0 = Completed.
✓ Exit Code non-zero = Error.
✓ Kubelet decides whether to restart a container.
✓ CrashLoopBackOff is a restart backoff state, not a restart policy.
✓ command specifies the executable.
✓ args specifies arguments.
✓ command overrides the image ENTRYPOINT.
✓ args overrides the image CMD.
✓ Images are read-only; Kubernetes never modifies an image.
✓ Containerd reads image metadata and combines it with kubelet's Pod configuration before starting the container.
------------------------------------------------------------
10. MY MENTAL MODEL
------------------------------------------------------------
Whenever I troubleshoot a Pod, I ask these questions:
1. What is the main process?
2. Is the process still running?
3. If it exited:
- What is the Exit Code?
4. What is the restartPolicy?
5. Is kubelet restarting the container?
6. Is the container in CrashLoopBackOff?
7. What executable (command) is Kubernetes starting?
8. What arguments (args) are being passed?
If I can answer these questions, I can usually determine why a Pod is in Running, Completed, Error, or CrashLoopBackOff.
A Pod gets its own network namespace.
Inside a namespace:
are isolated from the host.
Think:
Host Network Namespace ↓ Pod Network Namespace
A veth pair is like a virtual cable.
Example:
Pod eth0 ↔ veth123
One end is inside Pod.
One end is on host.
Traffic enters/leaves Pod through veth.
A bridge works like a virtual switch.
Example:
PodA ↓ veth ↓ Bridge ↓ veth ↓ PodB
Bridge works at Layer 2.
Routing decides:
"Where should packet go next?"
Example:
DST=172.16.212.50
Kernel checks routing table.
Chooses interface.
Sends packet.
Routing happens after DNAT.
Hooks are checkpoints inside kernel.
Packet just arrived.
Used mainly for:
Packet destined for local host.
Examples:
Packet passing through host.
Examples:
Packet generated locally.
Examples:
Packet leaving system.
Used mainly for:
Change destination.
Example:
10.96.0.25 ↓ 172.16.212.50
Used by:
Change source.
Example:
172.16.212.50 ↓ 192.168.241.141
Used when returning traffic.
Dynamic SNAT.
Commonly used when Pods access outside world.
Conntrack tracks connections.
Stores:
SRC IP SRC PORT DST IP DST PORT Protocol
Example:
192.168.1.10:50000 ↓ 10.96.0.25:80
States:
NEW ESTABLISHED RELATED INVALID
Purpose:
A Service is NOT:
Service is:
A virtual IP + kube-proxy rules.
Example:
rabbitmq-service
10.96.0.25
Flow:
Pod ↓ 10.96.0.25 ↓ kube-proxy ↓ DNAT ↓ RabbitMQ Pod
Watches:
from API Server.
Creates:
Main Job:
Service IP ↓ Pod IP
Example:
10.96.0.25 ↓ 172.16.212.50
Purpose:
Name ↓ Service IP
Example:
rabbitmq ↓ 10.96.0.25
Pod resolv.conf:
nameserver 10.96.0.10
Flow:
Application ↓ DNS Query ↓ 10.96.0.10 ↓ kube-proxy ↓ CoreDNS Pod ↓ Returns Service IP
CoreDNS watches API Server.
Keeps service records in memory.
Example:
rabbitmq → 10.96.0.25
mysql → 10.96.0.50
Purpose:
Calico uses Linux networking.
Creates:
Traffic:
PodA ↓ Host Network ↓ FORWARD ↓ Calico Rules ↓ PodB
Network Policy controls:
pod communication.
Implemented by Calico.
Calico creates iptables rules.
Example:
Allow:
frontend ↓ backend
Deny:
all others
Chain = Collection of Rules.
Example:
FORWARD chain
contains many rules.
Can call another chain.
Example:
FORWARD ↓ CALICO-FORWARD ↓ Policy Rules
Think:
Main Chain ↓ Sub Chain ↓ Rules
PodA ↓ veth ↓ Host Routing ↓ Calico ↓ Worker2 ↓ PodB
PodA ↓ Service IP 10.96.0.25 ↓ kube-proxy DNAT ↓ PodB
Application
curl rabbitmq
↓
CoreDNS
↓
10.96.0.25
↓
Application connects
↓
kube-proxy
↓
RabbitMQ Pod
Ingress Object:
Routing Rules.
Example:
myrabbitmqui.com ↓ rabbitmq-service
Ingress Object does NOT receive traffic.
Actual software.
Examples:
Receives HTTP/HTTPS traffic.
Reads:
Host Header
Example:
Host: myrabbitmqui.com
↓
rabbitmq-service
Browser ↓ 443 ↓ Ingress Controller ↓ Certificate Validation ↓ Encrypted Traffic
Certificate usually stored in:
Kubernetes Secret
Browser ↓ myrabbitmqui.com ↓ DNS ↓ Worker IP / LoadBalancer ↓ Ingress Controller ↓ rabbitmq-service ↓ kube-proxy ↓ RabbitMQ Pod
Name → Service IP
Service IP → Pod IP
Can packet travel there?
How packet reaches destination
Host Name → Service
Connection Tracking
Change Destination
Change Source
Virtual IP
Actual Application
Kubernetes networking is not a separate networking stack.
Kubernetes uses Linux networking.
Everything eventually becomes:
Packet → Linux Kernel → Routing → Netfilter → Socket → Application
Kubernetes components such as CoreDNS, kube-proxy, Calico, Services, and Ingress only add rules and automation on top of Linux networking.
Suppose a packet arrives from the network cable.
The Network Interface Card (NIC) receives the Ethernet frame.
The NIC places the frame into its RX Ring Buffer. This buffer is memory that allows packets to be temporarily stored until the kernel can process them.
The NIC uses DMA (Direct Memory Access) to copy the packet directly into RAM without involving the CPU. This improves performance because the CPU does not need to manually move every packet.
After copying the packet to memory, the NIC notifies the kernel.
Modern Linux uses NAPI instead of processing every packet immediately. NAPI allows packets to be processed in batches, preventing interrupt storms during high traffic.
The kernel then creates an SKB (Socket Buffer).
The SKB is the most important structure in Linux networking.
An SKB contains:
Think of an SKB as the kernel's internal representation of a packet.
The kernel first examines the Ethernet header.
The Ethernet header contains:
The EtherType tells Linux what protocol is inside the frame.
Examples:
0x0800 = IPv4 0x86DD = IPv6 0x0806 = ARP
If EtherType indicates IPv4, Linux passes the packet to the IP layer.
At the IP layer Linux examines:
Source IP Destination IP
Example:
Source IP = 192.168.1.10 Destination IP = 192.168.241.141
Before Linux decides where to send the packet, Netfilter hooks are executed.
Netfilter is a framework inside the Linux kernel.
It allows firewall, NAT and packet modification logic.
The five major hooks are:
PREROUTING INPUT FORWARD OUTPUT POSTROUTING
This hook runs before Linux makes a routing decision.
Typical uses:
Example:
Packet arrives:
Destination = 10.96.0.25
kube-proxy may change:
10.96.0.25
to
172.16.212.50
before routing happens.
INPUT is used when the packet is meant for the local machine itself.
Examples:
SSH Destination Port 22
Kubelet Destination Port 10250
The packet eventually reaches a socket owned by a process running on the host.
FORWARD is used when the packet is not for the host.
The host acts like a router.
Examples:
PodA → PodB
NodePort → Pod
Ingress Controller → Application Pod
Most Kubernetes traffic goes through FORWARD.
Used for packets created by the host itself.
Examples:
curl from the node
kubelet talking to API Server
Executed after routing decision and immediately before the packet leaves the system.
Used mainly for:
SNAT MASQUERADE
Routing determines where a packet should go.
Linux checks:
Destination IP
and consults:
Routing Table
The routing table tells Linux:
Which interface should carry the packet.
Which next hop should be used.
Routing happens after PREROUTING.
This is why DNAT must happen before routing.
If the destination IP changes, Linux needs the new destination before choosing the route.
NAT means modifying addresses.
There are three major types.
Destination NAT changes the destination address.
Example:
Destination = 10.96.0.25
becomes
Destination = 172.16.212.50
Used by:
Services NodePort Ingress
Source NAT changes the source address.
Example:
Source = 172.16.212.50
becomes
Source = 192.168.241.141
Used when traffic leaves the node.
MASQUERADE is dynamic SNAT.
Used when the external IP can change.
Commonly used when Pods access the Internet.
Conntrack is the kernel's connection tracking database.
It remembers:
Source IP Source Port Destination IP Destination Port Protocol
Suppose:
172.16.212.50:50000
connects to
10.96.0.25:80
After DNAT:
10.96.0.25
becomes
172.16.212.10
Conntrack remembers this translation.
When the response returns, conntrack ensures the reverse translation happens correctly.
Without conntrack, NAT would not work reliably.
Every Pod gets its own Network Namespace.
Inside the namespace the Pod has:
To connect the Pod to the host, Linux creates a veth pair.
Think of a veth pair as a virtual Ethernet cable.
One side exists inside the Pod.
The other side exists on the host.
Traffic leaving the Pod travels through this virtual cable.
A Service is not an application.
A Service is not a network interface.
A Service is not a process.
A Service is a virtual IP managed by kube-proxy.
Example:
rabbitmq-service
ClusterIP:
10.96.0.25
No application actually listens on 10.96.0.25.
kube-proxy intercepts packets destined for 10.96.0.25 and translates them to a real Pod IP.
kube-proxy watches the API Server.
It learns:
Services Endpoints
It then creates iptables rules.
Example:
10.96.0.25
becomes
172.16.212.50
This translation is performed using DNAT.
Therefore the primary job of kube-proxy is:
Service IP → Pod IP
Applications prefer names rather than IP addresses.
Example:
rabbitmq
instead of
10.96.0.25
CoreDNS watches the API Server and maintains DNS records in memory.
Example:
rabbitmq
→
10.96.0.25
When a Pod performs DNS lookup, CoreDNS returns the Service IP.
CoreDNS performs:
Name → Service IP
kube-proxy performs:
Service IP → Pod IP
Calico provides Pod networking and Network Policies.
Calico does not forward packets itself.
Linux performs the forwarding.
Calico programs:
Routes iptables rules Network Policies
Think:
Linux does the work.
Calico tells Linux what rules to use.
An Ingress object is only a routing rule.
Example:
myrabbitmqui.com
→
rabbitmq-service
The Ingress object itself receives no traffic.
An Ingress Controller such as NGINX reads the Ingress object and performs the routing.
NGINX receives the request, examines the Host header, and chooses the correct Service.
Browser opens:
https://myrabbitmqui.com
DNS returns:
192.168.241.141
Packet arrives at Worker Node.
NIC receives frame.
DMA copies frame into RAM.
NAPI schedules packet processing.
Kernel creates SKB.
Ethernet layer processes frame.
IP layer processes packet.
PREROUTING executes.
kube-proxy translates WorkerIP:443 to Ingress Controller Pod IP.
Routing decision occurs.
Packet traverses FORWARD chain.
Calico policy is checked.
Packet reaches NGINX Ingress Controller.
NGINX reads:
Host: myrabbitmqui.com
NGINX chooses:
rabbitmq-service
NGINX sends a new request to:
10.96.0.25
PREROUTING executes again.
kube-proxy translates:
10.96.0.25
to
172.16.212.50
Routing occurs.
Packet traverses FORWARD chain.
Packet reaches RabbitMQ Pod.
TCP delivers the packet to RabbitMQ's socket.
RabbitMQ application processes the request and sends the response back.
###############################
# MYSQL 8.0.x → 8.4.x UPGRADE
# CLEAN INSTALL + RESTORE METHOD
###############################
ENVIRONMENT:
- RHEL8 / Rocky Linux
- Custom datadir: /data/mysql_server
- No GTID
- Old MySQL 8.0.x
- New MySQL 8.4.x
- Using encryption/keyring
- Fresh initialize method
====================================================
STEP 1 — CHECK OLD MYSQL ENVIRONMENT
====================================================
mysql -uroot -p
SHOW PLUGINS;
SELECT TABLE_SCHEMA,TABLE_NAME,CREATE_OPTIONS
FROM information_schema.tables
WHERE CREATE_OPTIONS LIKE '%ENCRYPTION%';
grep -Ri keyring /etc/my.cnf*
====================================================
STEP 2 — TAKE FULL BACKUP
====================================================
mysqldump \
--all-databases \
--routines \
--events \
--triggers \
--single-transaction \
--hex-blob \
-u root -p > /backup/full.sql
====================================================
STEP 3 — BACKUP KEYRING (VERY IMPORTANT)
====================================================
tar -cvzf /backup/mysql-keyring.tar.gz \
/data/mysql-keyring
====================================================
STEP 4 — BACKUP CONFIG
====================================================
cp -p /etc/my.cnf /backup/
====================================================
STEP 5 — REMOVE ENCRYPTION FROM DUMP
(RECOMMENDED SAFEST METHOD)
====================================================
cp /backup/full.sql /backup/full_no_encrypt.sql
sed -i "s/ENCRYPTION='Y'/ENCRYPTION='N'/g" \
/backup/full_no_encrypt.sql
====================================================
STEP 6 — STOP MYSQL
====================================================
systemctl stop mysqld
ps -ef | grep mysqld
IF STILL RUNNING:
pkill -9 mysqld
====================================================
STEP 7 — REMOVE OLD MYSQL RPMs
====================================================
rpm -qa | grep -i mysql
dnf remove mysql*
OR
rpm -e mysql-community-server \
mysql-community-client \
mysql-community-common \
mysql-community-libs
====================================================
STEP 8 — INSTALL MYSQL 8.4 RPMs
====================================================
cd /data/pkg/mysql84/
yum install mysql-community-*.rpm
OR
rpm -ivh mysql-community-common-8.4*.rpm
rpm -ivh mysql-community-client-plugins-8.4*.rpm
rpm -ivh mysql-community-libs-8.4*.rpm
rpm -ivh mysql-community-client-8.4*.rpm
rpm -ivh mysql-community-server-8.4*.rpm
====================================================
STEP 9 — RENAME OLD DATADIR
====================================================
mv /data/mysql_server \
/data/mysql_server_80_backup
====================================================
STEP 10 — CREATE NEW DATADIR
====================================================
mkdir -p /data/mysql_server
chown -R mysql:mysql /data/mysql_server
chmod 750 /data/mysql_server
====================================================
STEP 11 — EDIT /etc/my.cnf
====================================================
REMOVE OLD KEYRING CONFIG:
#early-plugin-load=keyring_file.so
#keyring_file_data=/data/mysql-keyring/keyring
====================================================
STEP 12 — CREATE NEW KEYRING DIRECTORY
====================================================
mkdir -p /data/mysql-keyring
chown -R mysql:mysql /data/mysql-keyring
chmod 750 /data/mysql-keyring
====================================================
STEP 13 — CREATE COMPONENT CONFIG
====================================================
mkdir -p /var/lib/mysql-files
vi /var/lib/mysql-files/component_keyring_file.cnf
ADD:
{
"path": "/data/mysql-keyring/keyring",
"read_only": false
}
SAVE FILE
====================================================
STEP 14 — FIX PERMISSIONS
====================================================
chown mysql:mysql \
/var/lib/mysql-files/component_keyring_file.cnf
chmod 640 \
/var/lib/mysql-files/component_keyring_file.cnf
====================================================
STEP 15 — CREATE BOOTSTRAP FILE
(VERY IMPORTANT)
====================================================
vi /usr/sbin/mysqld.my
ADD:
{
"components": "file://component_keyring_file"
}
SAVE FILE
====================================================
STEP 16 — FIX BOOTSTRAP FILE PERMISSIONS
====================================================
chown mysql:mysql /usr/sbin/mysqld.my
chmod 640 /usr/sbin/mysqld.my
====================================================
STEP 17 — INITIALIZE MYSQL 8.4
====================================================
mysqld \
--defaults-file=/etc/my.cnf \
--initialize \
--user=mysql
====================================================
STEP 18 — START MYSQL
====================================================
systemctl start mysqld
====================================================
STEP 19 — CHECK LOGS
====================================================
journalctl -xeu mysqld
tail -f /data/mysql_server/mysqld.log
====================================================
STEP 20 — LOGIN MYSQL
====================================================
mysql -uroot -p
====================================================
STEP 21 — VERIFY KEYRING COMPONENT
====================================================
SELECT * FROM performance_schema.keyring_component_status;
IF NOT EMPTY = SUCCESS
====================================================
STEP 22 — REGISTER COMPONENT
====================================================
INSTALL COMPONENT 'file://component_keyring_file';
SELECT * FROM mysql.component;
====================================================
STEP 23 — TEST ENCRYPTION
====================================================
CREATE DATABASE testdb;
USE testdb;
CREATE TABLE t1 (
id INT
) ENCRYPTION='Y';
====================================================
STEP 24 — RESTORE DUMP
====================================================
mysql -uroot -p < /backup/full_no_encrypt.sql
====================================================
STEP 25 — VERIFY DATABASES
====================================================
SHOW DATABASES;
SELECT user,host FROM mysql.user;
====================================================
STEP 26 — OPTIONAL RE-ENABLE ENCRYPTION LATER
====================================================
ALTER TABLE table_name ENCRYPTION='Y';
====================================================
IMPORTANT NOTES
====================================================
1. NEVER DELETE:
/data/mysql-keyring
2. NEVER MIX:
old plugin + new component
3. DO NOT USE:
early-plugin-load=keyring_file.so
4. NEW MYSQL 8.4 USES:
component_keyring_file
5. MOST IMPORTANT FILE:
/usr/sbin/mysqld.my
WITHOUT IT:
- component installs
- but encryption fails
6. IF RESTORE FAILS:
- use ENCRYPTION='N'
- restore first
- re-enable later
###############################
END
###############################