# 🚀 RabbitMQ Cluster on Kubernetes (Complete Setup + Troubleshooting Guide)
---
# 📌 Objective
Deploy a **3-node RabbitMQ Cluster** on Kubernetes with:
* High Availability
* Persistent Storage (NFS)
* Auto Clustering
* Management UI
* Application connectivity (Tomcat)
---
# 🏗️ Components Created
## 1. Persistent Volumes (NFS)
We created 3 PVs:
* pv-rabbitmq1
* pv-rabbitmq2
* pv-rabbitmq3
Each mapped to:
```text
/data/nfsshared/rabbitmq-pv1
/data/nfsshared/rabbitmq-pv2
/data/nfsshared/rabbitmq-pv3
```
Used:
```yaml
accessModes: ReadWriteOnce
```
👉 Ensures **1 pod = 1 storage**
---
## 2. ConfigMap
Contains:
### enabled_plugins
```erlang
[rabbitmq_management,rabbitmq_peer_discovery_k8s].
```
### rabbitmq.conf
```ini
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = service-rabbitmq-headless
cluster_formation.k8s.hostname_suffix = .service-rabbitmq-headless.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
queue_master_locator=min-masters
```
👉 Enables **auto clustering using Kubernetes**
---
## 3. RBAC (CRITICAL)
```yaml
ServiceAccount → rabbitmq
Role → access pods, endpoints
RoleBinding → bind both
```
👉 Required because:
```text
RabbitMQ calls Kubernetes API → needs permission
```
---
## 4. Headless Service
```yaml
name: service-rabbitmq-headless
clusterIP: None
publishNotReadyAddresses: true
```
👉 Enables DNS like:
```text
rabbitmq-0.service-rabbitmq-headless
```
---
## 5. NodePort Service (UI)
```yaml
port: 15672
nodePort: 30072
```
👉 Access UI:
```text
http://<NodeIP>:30072
```
---
## 6. ClusterIP Service (App)
```yaml
name: rabbitmq-svc
port: 5672
```
👉 Used by:
```text
Tomcat → rabbitmq-svc:5672
```
---
## 7. StatefulSet
Key points:
```yaml
serviceName: service-rabbitmq-headless
replicas: 3
```
### ENV:
```yaml
RABBITMQ_DEFAULT_USER=admin
RABBITMQ_DEFAULT_PASS=admin
RABBITMQ_ERLANG_COOKIE=mysecretcookie
RABBITMQ_USE_LONGNAME=true
```
### Volumes:
* PVC → /var/lib/rabbitmq
* ConfigMap → rabbitmq.conf + plugins
👉 Ensures:
* Stable identity
* Persistent data
* Config-driven clustering
---
# ⚙️ FINAL EXECUTION ORDER (VERY IMPORTANT)
👉 Always follow this order:
```bash
kubectl apply -f pv-rabbit-01.yaml
kubectl apply -f pv-rabbit-02.yaml
kubectl apply -f pv-rabbit-03.yaml
kubectl apply -f rbac-rabbitmq.yaml
kubectl apply -f configmap-rabbit.yaml
kubectl apply -f service-rabbitmq-headless.yaml
kubectl apply -f service-rabbitmq-svc.yaml
kubectl apply -f service-rabbitmq-nodeport.yaml
kubectl apply -f StatefulSet-rabbitmq.yaml
```
---
# 🔥 TROUBLESHOOTING JOURNEY
---
## ❌ Issue 1: DNS Not Working
Problem:
```text
rabbitmq-1 not resolving
```
Fix:
```yaml
publishNotReadyAddresses: true
```
---
## ❌ Issue 2: Service Name Mismatch
Problem:
```text
rabbitmq-headless vs service-rabbitmq-headless
```
Fix:
```text
Must match EXACTLY
```
---
## ❌ Issue 3: No rabbitmq.conf
Fix:
Added clustering config
---
## ❌ Issue 4: 403 Error (CRITICAL)
Log:
```text
Failed to fetch nodes from Kubernetes API: 403
```
Fix:
Added RBAC
---
## ❌ Issue 5: Short vs Long Names
Error:
```text
epmd nxdomain
```
Fix:
```yaml
RABBITMQ_USE_LONGNAME=true
```
---
## ❌ Issue 6: Cluster Join Failure
Error:
```text
tables_not_present
mnesia_not_running
```
👉 Root cause:
```text
Pods not ready at same time (timing issue)
```
---
## ❌ Issue 7: Cluster Not Forming
Final log:
```text
Starting as a blank standalone node
```
👉 Reason:
```text
Retry failed → node becomes standalone
```
---
# 🧠 WHY THIS HAPPENS
RabbitMQ:
```text
Cluster formation happens ONLY at startup
```
If peers not ready → join fails
---
# 🔧 FINAL FIXES APPLIED
* Enabled RBAC ✅
* Enabled longnames ✅
* Fixed serviceName ✅
* Fixed DNS ✅
* Added retry logic ✅
* Restarted pods cleanly ✅
---
# 📊 FINAL RESULT
```bash
rabbitmqctl cluster_status
```
Output:
```text
rabbit@rabbitmq-0
rabbit@rabbitmq-1
rabbit@rabbitmq-2
```
---
# 🎯 WHAT WE ACHIEVED
✅ 3-node RabbitMQ cluster
✅ Auto discovery via Kubernetes
✅ Persistent storage
✅ UI access
✅ App connectivity
✅ HA-ready setup
---
# ⚠️ ALTERNATIVES
| Approach | Result |
| -------------- | ------------------------ |
| No RBAC | No clustering ❌ |
| Manual join | Works but not stable ⚠️ |
| Classic config | Static, not scalable ❌ |
| Helm chart | Best production option ✅ |
---
# 🧠 FINAL LEARNING
* Kubernetes = dynamic → needs API
* RabbitMQ = startup-based clustering
* RBAC = mandatory
* Headless service = must
* Longnames = required
* Timing = critical
---
# 🚀 NEXT STEPS
* Create quorum queues
* Test failover (kill pod)
* Connect Tomcat producer/consumer
* Monitor cluster
---
# 📌 FINAL CONCLUSION
You successfully built a **production-grade RabbitMQ cluster on Kubernetes**
and solved real-world issues like:
* DNS
* RBAC
* Clustering
* Node naming
* Startup timing
---
==================================configmap-rabbit.yaml========================================
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap-rabbit
labels:
type: configmap-rabbit
data:
enabled_plugins: |
[rabbitmq_management,rabbitmq_peer_discovery_k8s].
rabbitmq.conf: |
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = service-rabbitmq-headless
cluster_formation.k8s.hostname_suffix = .service-rabbitmq-headless.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
queue_master_locator=min-masters
==================================pv-rabbit-01.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq1
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv1
readOnly: false
==================================pv-rabbit-02.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq2
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv2
readOnly: false
==================================pv-rabbit-03.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq3
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv3
readOnly: false
==================================rbac-rabbitmq.yaml========================================
apiVersion: v1
kind: ServiceAccount
metadata:
name: rabbitmq
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rabbitmq
namespace: default
rules:
- apiGroups: [""]
resources:
- endpoints
- pods
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rabbitmq
namespace: default
subjects:
- kind: ServiceAccount
name: rabbitmq
namespace: default
roleRef:
kind: Role
name: rabbitmq
apiGroup: rbac.authorization.k8s.io
==================================service-rabbitmq-headless.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: service-rabbitmq-headless
labels:
type: service-rabbitmq-headless
spec:
clusterIP: None
publishNotReadyAddresses: true
selector:
app: rabbitmq
ports:
- name: amqp
port: 5672
- name: management
port: 15672
- name: epmd
port: 4369
- name: cluster-rpc
port: 25672
==================================service-rabbitmq-nodeport.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-nodeport
labels:
type: rabbitmq-nodeport
spec:
type: NodePort
selector:
app: rabbitmq
ports:
- name: management
port: 15672
targetPort: 15672
nodePort: 30072
==================================service-rabbitmq-svc.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-svc
labels:
type: rabbitmq-svc
spec:
type: ClusterIP
selector:
app: rabbitmq
ports:
- name: amqp
port: 5672
targetPort: 5672
==================================StatefulSet-rabbitmq.yaml========================================
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rabbitmq
labels:
type: rabbitmq
spec:
serviceName: service-rabbitmq-headless
replicas: 3
selector:
matchLabels:
app: rabbitmq
template:
metadata:
labels:
app: rabbitmq
spec:
serviceAccountName: rabbitmq
containers:
- name: rabbitmq
image: rabbitmq:3.12-management
ports:
- containerPort: 5672
- containerPort: 15672
env:
- name: RABBITMQ_DEFAULT_USER
value: "admin"
- name: RABBITMQ_DEFAULT_PASS
value: "admin"
- name: RABBITMQ_ERLANG_COOKIE
value: "mysecretcookie"
- name: RABBITMQ_USE_LONGNAME
value: "true"
volumeMounts:
- name: data
mountPath: /var/lib/rabbitmq
- name: config
mountPath: /etc/rabbitmq/enabled_plugins
subPath: enabled_plugins
- name: rabbitconf
mountPath: /etc/rabbitmq/rabbitmq.conf
subPath: rabbitmq.conf
volumes:
- name: config
configMap:
name: configmap-rabbit
- name: rabbitconf
configMap:
name: configmap-rabbit
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: ""
resources:
requests:
storage: 1Gi
selector:
matchLabels:
type: pv-rabbitmq
# 👍 END
| Task | Imperative Command | Declarative YAML |
| Create Namespace | kubectl create namespace dev | apiVersion: v1 kind: Namespace metadata: name: dev |
| Create Pod | kubectl run nginx-pod --image=nginx:1.25 | apiVersion: v1 kind: Pod metadata: name: nginx-pod spec: containers: - name: nginx image: nginx:1.25 |
| Create Deployment | kubectl create deployment sampledeploy --image=nginx:1.25 --replicas=4 | apiVersion: apps/v1 kind: Deployment metadata: name: sampledeploy spec: replicas: 4 selector: matchLabels: app: sampledeploy template: metadata: labels: app: sampledeploy spec: containers: - name: nginx image: nginx:1.25 |
| Create ClusterIP Service | kubectl expose deployment sampledeploy --port=80 --target-port=80 | apiVersion: v1 kind: Service metadata: name: sampledeploy-service spec: selector: app: sampledeploy ports: - port: 80 targetPort: 80 |
| Create NodePort Service | kubectl expose deployment sampledeploy --type=NodePort --port=80 --target-port=80 | apiVersion: v1 kind: Service metadata: name: sampledeploy-nodeport spec: type: NodePort selector: app: sampledeploy ports: - port: 80 targetPort: 80 nodePort: 30080 |
| Create ConfigMap | kubectl create configmap app-config --from-literal=APP_MODE=production --from-literal=COLOR=blue | apiVersion: v1 kind: ConfigMap metadata: name: app-config data: APP_MODE: production COLOR: blue |
| Create Secret | kubectl create secret generic db-secret --from-literal=username=admin --from-literal=password=pass123 | apiVersion: v1 kind: Secret metadata: name: db-secret type: Opaque stringData: username: admin password: pass123 |
| Create ServiceAccount | kubectl create serviceaccount frontend-sa | apiVersion: v1 kind: ServiceAccount metadata: name: frontend-sa |
| Create Role | kubectl create role pod-reader --verb=get,list,watch --resource=pods | apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] |
| Create RoleBinding | kubectl create rolebinding pod-reader-binding --role=pod-reader --serviceaccount=default:frontend-sa | apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pod-reader-binding subjects: - kind: ServiceAccount name: frontend-sa namespace: default roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io |
| Create ClusterRole | kubectl create clusterrole node-reader --verb=get,list,watch --resource=nodes | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: node-reader rules: - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] |
| Create ClusterRoleBinding | kubectl create clusterrolebinding node-reader-binding --clusterrole=node-reader --serviceaccount=default:frontend-sa | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-reader-binding subjects: - kind: ServiceAccount name: frontend-sa namespace: default roleRef: kind: ClusterRole name: node-reader apiGroup: rbac.authorization.k8s.io |
| Create PVC | Usually declarative | apiVersion: v1 kind: PersistentVolumeClaim metadata: name: app-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi |
| Create StatefulSet | Usually declarative | apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb spec: serviceName: mongodb-headless replicas: 1 selector: matchLabels: app: mongodb template: metadata: labels: app: mongodb spec: containers: - name: mongodb image: mongo:6 |
| Create Job | kubectl create job test-job --image=busybox | apiVersion: batch/v1 kind: Job metadata: name: test-job spec: template: spec: containers: - name: busybox image: busybox restartPolicy: Never |
| Create CronJob | kubectl create cronjob backup-job --image=busybox --schedule="*/5 * * * *" | apiVersion: batch/v1 kind: CronJob metadata: name: backup-job spec: schedule: "*/5 * * * *" jobTemplate: spec: template: spec: containers: - name: busybox image: busybox restartPolicy: Never |
###############################
# MYSQL 8.0.x → 8.4.x UPGRADE
# CLEAN INSTALL + RESTORE METHOD
###############################
ENVIRONMENT:
- RHEL8 / Rocky Linux
- Custom datadir: /data/mysql_server
- No GTID
- Old MySQL 8.0.x
- New MySQL 8.4.x
- Using encryption/keyring
- Fresh initialize method
====================================================
STEP 1 — CHECK OLD MYSQL ENVIRONMENT
====================================================
mysql -uroot -p
SHOW PLUGINS;
SELECT TABLE_SCHEMA,TABLE_NAME,CREATE_OPTIONS
FROM information_schema.tables
WHERE CREATE_OPTIONS LIKE '%ENCRYPTION%';
grep -Ri keyring /etc/my.cnf*
====================================================
STEP 2 — TAKE FULL BACKUP
====================================================
mysqldump \
--all-databases \
--routines \
--events \
--triggers \
--single-transaction \
--hex-blob \
-u root -p > /backup/full.sql
====================================================
STEP 3 — BACKUP KEYRING (VERY IMPORTANT)
====================================================
tar -cvzf /backup/mysql-keyring.tar.gz \
/data/mysql-keyring
====================================================
STEP 4 — BACKUP CONFIG
====================================================
cp -p /etc/my.cnf /backup/
====================================================
STEP 5 — REMOVE ENCRYPTION FROM DUMP
(RECOMMENDED SAFEST METHOD)
====================================================
cp /backup/full.sql /backup/full_no_encrypt.sql
sed -i "s/ENCRYPTION='Y'/ENCRYPTION='N'/g" \
/backup/full_no_encrypt.sql
====================================================
STEP 6 — STOP MYSQL
====================================================
systemctl stop mysqld
ps -ef | grep mysqld
IF STILL RUNNING:
pkill -9 mysqld
====================================================
STEP 7 — REMOVE OLD MYSQL RPMs
====================================================
rpm -qa | grep -i mysql
dnf remove mysql*
OR
rpm -e mysql-community-server \
mysql-community-client \
mysql-community-common \
mysql-community-libs
====================================================
STEP 8 — INSTALL MYSQL 8.4 RPMs
====================================================
cd /data/pkg/mysql84/
yum install mysql-community-*.rpm
OR
rpm -ivh mysql-community-common-8.4*.rpm
rpm -ivh mysql-community-client-plugins-8.4*.rpm
rpm -ivh mysql-community-libs-8.4*.rpm
rpm -ivh mysql-community-client-8.4*.rpm
rpm -ivh mysql-community-server-8.4*.rpm
====================================================
STEP 9 — RENAME OLD DATADIR
====================================================
mv /data/mysql_server \
/data/mysql_server_80_backup
====================================================
STEP 10 — CREATE NEW DATADIR
====================================================
mkdir -p /data/mysql_server
chown -R mysql:mysql /data/mysql_server
chmod 750 /data/mysql_server
====================================================
STEP 11 — EDIT /etc/my.cnf
====================================================
REMOVE OLD KEYRING CONFIG:
#early-plugin-load=keyring_file.so
#keyring_file_data=/data/mysql-keyring/keyring
====================================================
STEP 12 — CREATE NEW KEYRING DIRECTORY
====================================================
mkdir -p /data/mysql-keyring
chown -R mysql:mysql /data/mysql-keyring
chmod 750 /data/mysql-keyring
====================================================
STEP 13 — CREATE COMPONENT CONFIG
====================================================
mkdir -p /var/lib/mysql-files
vi /var/lib/mysql-files/component_keyring_file.cnf
ADD:
{
"path": "/data/mysql-keyring/keyring",
"read_only": false
}
SAVE FILE
====================================================
STEP 14 — FIX PERMISSIONS
====================================================
chown mysql:mysql \
/var/lib/mysql-files/component_keyring_file.cnf
chmod 640 \
/var/lib/mysql-files/component_keyring_file.cnf
====================================================
STEP 15 — CREATE BOOTSTRAP FILE
(VERY IMPORTANT)
====================================================
vi /usr/sbin/mysqld.my
ADD:
{
"components": "file://component_keyring_file"
}
SAVE FILE
====================================================
STEP 16 — FIX BOOTSTRAP FILE PERMISSIONS
====================================================
chown mysql:mysql /usr/sbin/mysqld.my
chmod 640 /usr/sbin/mysqld.my
====================================================
STEP 17 — INITIALIZE MYSQL 8.4
====================================================
mysqld \
--defaults-file=/etc/my.cnf \
--initialize \
--user=mysql
====================================================
STEP 18 — START MYSQL
====================================================
systemctl start mysqld
====================================================
STEP 19 — CHECK LOGS
====================================================
journalctl -xeu mysqld
tail -f /data/mysql_server/mysqld.log
====================================================
STEP 20 — LOGIN MYSQL
====================================================
mysql -uroot -p
====================================================
STEP 21 — VERIFY KEYRING COMPONENT
====================================================
SELECT * FROM performance_schema.keyring_component_status;
IF NOT EMPTY = SUCCESS
====================================================
STEP 22 — REGISTER COMPONENT
====================================================
INSTALL COMPONENT 'file://component_keyring_file';
SELECT * FROM mysql.component;
====================================================
STEP 23 — TEST ENCRYPTION
====================================================
CREATE DATABASE testdb;
USE testdb;
CREATE TABLE t1 (
id INT
) ENCRYPTION='Y';
====================================================
STEP 24 — RESTORE DUMP
====================================================
mysql -uroot -p < /backup/full_no_encrypt.sql
====================================================
STEP 25 — VERIFY DATABASES
====================================================
SHOW DATABASES;
SELECT user,host FROM mysql.user;
====================================================
STEP 26 — OPTIONAL RE-ENABLE ENCRYPTION LATER
====================================================
ALTER TABLE table_name ENCRYPTION='Y';
====================================================
IMPORTANT NOTES
====================================================
1. NEVER DELETE:
/data/mysql-keyring
2. NEVER MIX:
old plugin + new component
3. DO NOT USE:
early-plugin-load=keyring_file.so
4. NEW MYSQL 8.4 USES:
component_keyring_file
5. MOST IMPORTANT FILE:
/usr/sbin/mysqld.my
WITHOUT IT:
- component installs
- but encryption fails
6. IF RESTORE FAILS:
- use ENCRYPTION='N'
- restore first
- re-enable later
###############################
END
###############################
Kubernetes Networking Notes
A Pod gets its own network namespace.
Inside a namespace:
are isolated from the host.
Think:
Host Network Namespace ↓ Pod Network Namespace
A veth pair is like a virtual cable.
Example:
Pod eth0 ↔ veth123
One end is inside Pod.
One end is on host.
Traffic enters/leaves Pod through veth.
A bridge works like a virtual switch.
Example:
PodA ↓ veth ↓ Bridge ↓ veth ↓ PodB
Bridge works at Layer 2.
Routing decides:
"Where should packet go next?"
Example:
DST=172.16.212.50
Kernel checks routing table.
Chooses interface.
Sends packet.
Routing happens after DNAT.
Hooks are checkpoints inside kernel.
Packet just arrived.
Used mainly for:
Packet destined for local host.
Examples:
Packet passing through host.
Examples:
Packet generated locally.
Examples:
Packet leaving system.
Used mainly for:
Change destination.
Example:
10.96.0.25 ↓ 172.16.212.50
Used by:
Change source.
Example:
172.16.212.50 ↓ 192.168.241.141
Used when returning traffic.
Dynamic SNAT.
Commonly used when Pods access outside world.
Conntrack tracks connections.
Stores:
SRC IP SRC PORT DST IP DST PORT Protocol
Example:
192.168.1.10:50000 ↓ 10.96.0.25:80
States:
NEW ESTABLISHED RELATED INVALID
Purpose:
A Service is NOT:
Service is:
A virtual IP + kube-proxy rules.
Example:
rabbitmq-service
10.96.0.25
Flow:
Pod ↓ 10.96.0.25 ↓ kube-proxy ↓ DNAT ↓ RabbitMQ Pod
Watches:
from API Server.
Creates:
Main Job:
Service IP ↓ Pod IP
Example:
10.96.0.25 ↓ 172.16.212.50
Purpose:
Name ↓ Service IP
Example:
rabbitmq ↓ 10.96.0.25
Pod resolv.conf:
nameserver 10.96.0.10
Flow:
Application ↓ DNS Query ↓ 10.96.0.10 ↓ kube-proxy ↓ CoreDNS Pod ↓ Returns Service IP
CoreDNS watches API Server.
Keeps service records in memory.
Example:
rabbitmq → 10.96.0.25
mysql → 10.96.0.50
Purpose:
Calico uses Linux networking.
Creates:
Traffic:
PodA ↓ Host Network ↓ FORWARD ↓ Calico Rules ↓ PodB
Network Policy controls:
pod communication.
Implemented by Calico.
Calico creates iptables rules.
Example:
Allow:
frontend ↓ backend
Deny:
all others
Chain = Collection of Rules.
Example:
FORWARD chain
contains many rules.
Can call another chain.
Example:
FORWARD ↓ CALICO-FORWARD ↓ Policy Rules
Think:
Main Chain ↓ Sub Chain ↓ Rules
PodA ↓ veth ↓ Host Routing ↓ Calico ↓ Worker2 ↓ PodB
PodA ↓ Service IP 10.96.0.25 ↓ kube-proxy DNAT ↓ PodB
Application
curl rabbitmq
↓
CoreDNS
↓
10.96.0.25
↓
Application connects
↓
kube-proxy
↓
RabbitMQ Pod
Ingress Object:
Routing Rules.
Example:
myrabbitmqui.com ↓ rabbitmq-service
Ingress Object does NOT receive traffic.
Actual software.
Examples:
Receives HTTP/HTTPS traffic.
Reads:
Host Header
Example:
Host: myrabbitmqui.com
↓
rabbitmq-service
Browser ↓ 443 ↓ Ingress Controller ↓ Certificate Validation ↓ Encrypted Traffic
Certificate usually stored in:
Kubernetes Secret
Browser ↓ myrabbitmqui.com ↓ DNS ↓ Worker IP / LoadBalancer ↓ Ingress Controller ↓ rabbitmq-service ↓ kube-proxy ↓ RabbitMQ Pod
Name → Service IP
Service IP → Pod IP
Can packet travel there?
How packet reaches destination
Host Name → Service
Connection Tracking
Change Destination
Change Source
Virtual IP
Actual Application
Linux To Kubernetes Networking - Complete Mental Model
Kubernetes networking is not a separate networking stack.
Kubernetes uses Linux networking.
Everything eventually becomes:
Packet → Linux Kernel → Routing → Netfilter → Socket → Application
Kubernetes components such as CoreDNS, kube-proxy, Calico, Services, and Ingress only add rules and automation on top of Linux networking.
Suppose a packet arrives from the network cable.
The Network Interface Card (NIC) receives the Ethernet frame.
The NIC places the frame into its RX Ring Buffer. This buffer is memory that allows packets to be temporarily stored until the kernel can process them.
The NIC uses DMA (Direct Memory Access) to copy the packet directly into RAM without involving the CPU. This improves performance because the CPU does not need to manually move every packet.
After copying the packet to memory, the NIC notifies the kernel.
Modern Linux uses NAPI instead of processing every packet immediately. NAPI allows packets to be processed in batches, preventing interrupt storms during high traffic.
The kernel then creates an SKB (Socket Buffer).
The SKB is the most important structure in Linux networking.
An SKB contains:
Think of an SKB as the kernel's internal representation of a packet.
The kernel first examines the Ethernet header.
The Ethernet header contains:
The EtherType tells Linux what protocol is inside the frame.
Examples:
0x0800 = IPv4 0x86DD = IPv6 0x0806 = ARP
If EtherType indicates IPv4, Linux passes the packet to the IP layer.
At the IP layer Linux examines:
Source IP Destination IP
Example:
Source IP = 192.168.1.10 Destination IP = 192.168.241.141
Before Linux decides where to send the packet, Netfilter hooks are executed.
Netfilter is a framework inside the Linux kernel.
It allows firewall, NAT and packet modification logic.
The five major hooks are:
PREROUTING INPUT FORWARD OUTPUT POSTROUTING
This hook runs before Linux makes a routing decision.
Typical uses:
Example:
Packet arrives:
Destination = 10.96.0.25
kube-proxy may change:
10.96.0.25
to
172.16.212.50
before routing happens.
INPUT is used when the packet is meant for the local machine itself.
Examples:
SSH Destination Port 22
Kubelet Destination Port 10250
The packet eventually reaches a socket owned by a process running on the host.
FORWARD is used when the packet is not for the host.
The host acts like a router.
Examples:
PodA → PodB
NodePort → Pod
Ingress Controller → Application Pod
Most Kubernetes traffic goes through FORWARD.
Used for packets created by the host itself.
Examples:
curl from the node
kubelet talking to API Server
Executed after routing decision and immediately before the packet leaves the system.
Used mainly for:
SNAT MASQUERADE
Routing determines where a packet should go.
Linux checks:
Destination IP
and consults:
Routing Table
The routing table tells Linux:
Which interface should carry the packet.
Which next hop should be used.
Routing happens after PREROUTING.
This is why DNAT must happen before routing.
If the destination IP changes, Linux needs the new destination before choosing the route.
NAT means modifying addresses.
There are three major types.
Destination NAT changes the destination address.
Example:
Destination = 10.96.0.25
becomes
Destination = 172.16.212.50
Used by:
Services NodePort Ingress
Source NAT changes the source address.
Example:
Source = 172.16.212.50
becomes
Source = 192.168.241.141
Used when traffic leaves the node.
MASQUERADE is dynamic SNAT.
Used when the external IP can change.
Commonly used when Pods access the Internet.
Conntrack is the kernel's connection tracking database.
It remembers:
Source IP Source Port Destination IP Destination Port Protocol
Suppose:
172.16.212.50:50000
connects to
10.96.0.25:80
After DNAT:
10.96.0.25
becomes
172.16.212.10
Conntrack remembers this translation.
When the response returns, conntrack ensures the reverse translation happens correctly.
Without conntrack, NAT would not work reliably.
Every Pod gets its own Network Namespace.
Inside the namespace the Pod has:
To connect the Pod to the host, Linux creates a veth pair.
Think of a veth pair as a virtual Ethernet cable.
One side exists inside the Pod.
The other side exists on the host.
Traffic leaving the Pod travels through this virtual cable.
A Service is not an application.
A Service is not a network interface.
A Service is not a process.
A Service is a virtual IP managed by kube-proxy.
Example:
rabbitmq-service
ClusterIP:
10.96.0.25
No application actually listens on 10.96.0.25.
kube-proxy intercepts packets destined for 10.96.0.25 and translates them to a real Pod IP.
kube-proxy watches the API Server.
It learns:
Services Endpoints
It then creates iptables rules.
Example:
10.96.0.25
becomes
172.16.212.50
This translation is performed using DNAT.
Therefore the primary job of kube-proxy is:
Service IP → Pod IP
Applications prefer names rather than IP addresses.
Example:
rabbitmq
instead of
10.96.0.25
CoreDNS watches the API Server and maintains DNS records in memory.
Example:
rabbitmq
→
10.96.0.25
When a Pod performs DNS lookup, CoreDNS returns the Service IP.
CoreDNS performs:
Name → Service IP
kube-proxy performs:
Service IP → Pod IP
Calico provides Pod networking and Network Policies.
Calico does not forward packets itself.
Linux performs the forwarding.
Calico programs:
Routes iptables rules Network Policies
Think:
Linux does the work.
Calico tells Linux what rules to use.
An Ingress object is only a routing rule.
Example:
myrabbitmqui.com
→
rabbitmq-service
The Ingress object itself receives no traffic.
An Ingress Controller such as NGINX reads the Ingress object and performs the routing.
NGINX receives the request, examines the Host header, and chooses the correct Service.
Browser opens:
https://myrabbitmqui.com
DNS returns:
192.168.241.141
Packet arrives at Worker Node.
NIC receives frame.
DMA copies frame into RAM.
NAPI schedules packet processing.
Kernel creates SKB.
Ethernet layer processes frame.
IP layer processes packet.
PREROUTING executes.
kube-proxy translates WorkerIP:443 to Ingress Controller Pod IP.
Routing decision occurs.
Packet traverses FORWARD chain.
Calico policy is checked.
Packet reaches NGINX Ingress Controller.
NGINX reads:
Host: myrabbitmqui.com
NGINX chooses:
rabbitmq-service
NGINX sends a new request to:
10.96.0.25
PREROUTING executes again.
kube-proxy translates:
10.96.0.25
to
172.16.212.50
Routing occurs.
Packet traverses FORWARD chain.
Packet reaches RabbitMQ Pod.
TCP delivers the packet to RabbitMQ's socket.
RabbitMQ application processes the request and sends the response back.