1) Buy a VPS Server
for setting up django web app in centos/Rhel vps system you have to buy vps server from vps provider it will give you server ip address and root password also you will get client web gui for management of server from that you get access to reset server ,reinstall server or boot sever into safe mode and from there you can even change the root password
2) ssh the server
now ssh the sever from any ssh client like putty ,cmd or mobxtrem,etc.
ssh root@192.168.47.101 ---> give root password of server
after that you will get the full access that server
3)Change the root password
once you login to ssh server you should change your root password by passwd commnad
passwd root --> give new password two times
4) add a new user
add a new user so you dont needed to use root for all the times it will make your server more secure
useradd test
passwd test --> give new password two times
now add that test user to wheel group so you can use sudo command by sudo command you can access all system command by that new user
usermod -G wheel test
5) Configure ssh
configure ssh so no one can login to the system via root , edit ssh configuration file and write PermitRootLogin to No
vim /etc/ssh/sshd_config
PermitRootLogin to No
now reboot the system and login via ssh by ssh user@server
6)update the system
now update the system then reboot the system
e.g yum update && reboot
7)Install python3
after sucessfully update install python3
yum install python3
8)install and activate virtual enviroment
after installation of python3 we can use python pip package installer by using pip we can install and create virtual enviroment
by using virtual enviroment we can install python packages virtually that means they only available when we activate the virtual enviroment
pip3 install virtualenv
9)activate virtual enviroment
for activation of virtual enviroment create new directory use source command to activate it
mkdir /djangoenv
virtualenv /djangoenv
source /django/bin/activate
10)install python packages
first create the requirement.txt ,in that file write list of names of python packages
now install it by
pip install -r requirements.txt
11) install sqlite3.8
for django to work in python3 in centos it need the sqlite version 3.8
wget https://kojipkgs.fedoraproject.org//packages/sqlite/3.8.11/1.fc21/x86_64/sqlite-3.8.11-1.fc21.x86_64.rpm
sudo yum install sqlite-3.8.11-1.fc21.x86_64.rpm
12)copy django application to server
copy django application to server by using scp or sftp
scp djangoapplication.zip user@server
13)test the django application
python3 manage.py runserver
14)install apache webserver
install apache webserver
yum install httpd*
15) configure apache server
go to /etc/httpd/conf.d
create web.conf file
type configuration
<virtual host *:80>
servername yoursitename
alias /static /your static directory in django application
alias /media /your media directory in django application
<directory /your static direcory >
require all granted
</directory>
<directory /your media directory >
require all granted
</directory>
<directory /your main app directory where wsgi file located>
<files wsgi.py>
require all granted
</files>
</directory>
WSGIDaemonProcess name_of_your_project python-path = /directory of project
python-home = /directory of enviroment variables'
WSGIProcessGroup name_of_project
WSGIScriptAlias / /your django app directory
</virtualhost>
save and exit
if there is numpy python package in your django app then it will gives time out so add following line to
/etc/httpd/conf/httpd.conf
WSGIApplicationGroup % {GLOBAL}
save and exit
16)Restart apache server
test apache config by
apachectl configtest
restart apache server
systemctl restart httpd
17)go to web browser check your site
Done.
Create Replication Mysql
mysqld --defaults-file=/etc/my.cnf --initialize --user=mysql
In 8.4 Community, the host cache is managed internally.
FLUSH HOST DEPRICATED --> SELECT * FROM performance_schema.host_cache;
CREATE USER 'repl'@'%' IDENTIFIED BY 'password';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';
FLUSH PRIVILEGES;
CHANGE REPLICATION SOURCE TO
SOURCE_HOST='192.168.241.101',
SOURCE_USER='repl',
SOURCE_PASSWORD='password',
SOURCE_PORT=3360,
SOURCE_AUTO_POSITION=1,
SOURCE_SSL=1,
SOURCE_SSL_CA='/data/ca.pem';
slave my.cnf
[mysqld]
datadir=/data/mysql_server
socket=/var/lib/mysql/mysql.sock
log-error=/data/mysql_server/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
server-id=2
log_bin=mysql-bin
binlog_format=ROW
gtid_mode=ON
enforce_gtid_consistency=ON
get_source_public_key=1
port=3360
user=mysql
symbolic-links=0
# Connection limits (safe for low-memory VM)
max_connections=50
max_user_connections=50
# Packet and temporary table sizes
max_allowed_packet=16M
tmp_table_size=32M
max_heap_table_size=32M
# Sorting and read buffers (per connection, smaller for low RAM)
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size=4M
join_buffer_size=2M
# Storage engine
default-storage-engine=InnoDB
key_buffer_size=8M
bulk_insert_buffer_size=8M
# InnoDB settings for small memory
innodb_log_file_size=32M
innodb_print_all_deadlocks=1
innodb_buffer_pool_instances=1
innodb_buffer_pool_size=512M
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_thread_concurrency=0
innodb_io_capacity=100
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=2
innodb_lock_wait_timeout=50
# Transaction isolation
transaction-isolation=READ-COMMITTED
[root@node2 data]#
============================
master my.cnf
[mysqld]
datadir=/data/mysql_server
socket=/var/lib/mysql/mysql.sock
log-error=/data/mysql_server/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
server-id=1
log_bin=mysql-bin
binlog_format=ROW
gtid_mode=ON
enforce_gtid_consistency=ON
plugin-load-add=mysql_native_password.so
port=3360
user=mysql
symbolic-links=0
# Connection limits (safe for low-memory VM)
max_connections=50
max_user_connections=50
# Packet and temporary table sizes
max_allowed_packet=16M
tmp_table_size=32M
max_heap_table_size=32M
# Sorting and read buffers (per connection, smaller for low RAM)
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size=4M
join_buffer_size=2M
# Storage engine
default-storage-engine=InnoDB
key_buffer_size=8M
bulk_insert_buffer_size=8M
# InnoDB settings for small memory
innodb_log_file_size=32M
innodb_print_all_deadlocks=1
innodb_buffer_pool_instances=1
innodb_buffer_pool_size=512M
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_thread_concurrency=0
innodb_io_capacity=100
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=2
innodb_lock_wait_timeout=50
# Transaction isolation
transaction-isolation=READ-COMMITTED
Secure Monitoring Setup – Prometheus + Node Exporter + Grafana (HTTPS + TLS + Django Embed)
OS: Rocky Linux 8.10 / RHEL compatible
Container runtime: Podman
TLS: Let’s Encrypt or self-signed
Grafana HTTPS port: 42923
Goal:
– node exporter & prometheus not public
– only localhost access
– grafana SSL enabled on random port
– disable anonymous login
– embed inside Django site
– force login authentication
=======================================
Create directories
mkdir -p /opt/node
mkdir -p /opt/prometheus
mkdir -p /opt/grafana/certs
=======================================
2) Copy SSL certificates for Grafana
Use Let’s Encrypt certs (recommended)
cp -L /etc/letsencrypt/live/www.yourdomain.com/fullchain.pem /opt/grafana/certs/fullchain.pem
cp -L /etc/letsencrypt/live/www.yourdomain.com/privkey.pem /opt/grafana/certs/privkey.pem
Fix permissions
chmod 640 /opt/grafana/certs/privkey.pem
chmod 644 /opt/grafana/certs/fullchain.pem
SELinux label
chcon -Rt container_file_t /opt/grafana/certs
=======================================
3) Create Node Exporter HTTPS config file
cat > /opt/node/web.yml <<EOF
tls_server_config:
cert_file: /certs/server.crt
key_file: /certs/server.key
EOF
If using self-signed:
copy cert files
cp /opt/ssl/selfsigned/server.crt /opt/node/
cp /opt/ssl/selfsigned/server.key /opt/node/
=======================================
4) Run Node Exporter (localhost only + HTTPS)
podman run -d
--name node_exporter
-p 127.0.0.1:9100:9100
-v /opt/node/web.yml:/etc/node/web.yml:Z
-v /opt/ssl/selfsigned/server.crt:/certs/server.crt:ro,Z
-v /opt/ssl/selfsigned/server.key:/certs/server.key:ro,Z
quay.io/prometheus/node-exporter
--web.config.file=/etc/node/web.yml
--web.listen-address=127.0.0.1:9100
Test
curl -k https://127.0.0.1:9100/metrics
=======================================
5) Create Prometheus config
cat > /opt/prometheus/prometheus.yml <<EOF
global:
scrape_interval: 5s
scrape_configs:
job_name: 'node'
scheme: https
tls_config:
insecure_skip_verify: true
static_configs:
targets: ['127.0.0.1:9100']
EOF
=======================================
6) Run Prometheus container
podman run -d
--name prometheus
-p 127.0.0.1:9090:9090
-v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:Z
prom/prometheus
Test local access
curl -k https://127.0.0.1:9090/api/v1/targets
You should see "health: up"
=======================================
7) Run Grafana HTTPS on port 42923
Remove any old container first
podman stop grafana || true
podman rm grafana || true
Run new
podman run -d
--name=grafana
--net=host
-v grafana:/var/lib/grafana
-v /opt/grafana/certs/fullchain.pem:/certs/fullchain.pem:ro,Z
-v /opt/grafana/certs/privkey.pem:/certs/privkey.pem:ro,Z
-e GF_SERVER_PROTOCOL=https
-e GF_SERVER_HTTP_PORT=42923
-e GF_SERVER_CERT_FILE=/certs/fullchain.pem
-e GF_SERVER_CERT_KEY=/certs/privkey.pem
-e GF_AUTH_ANONYMOUS_ENABLED=false
-e GF_USERS_ALLOW_SIGN_UP=false
-e GF_SECURITY_ALLOW_EMBEDDING=true
grafana/grafana:latest
Test
curl -k https://127.0.0.1:42923
Expected:
<a href="/login">Found</a>
Browser URL
https://www.yourdomain.com:42923
=======================================
8) First Grafana login
Default user:
admin / admin
Grafana will force you to CHANGE PASSWORD
Create user “xxxx” later in settings.
=======================================
9) Import dashboard ID 1860
In Grafana:
Dashboards → Import
Dashboard ID: 1860
Select Prometheus datasource
=======================================
10) Django embed setup
Create app: serverstats
views.py
from django.contrib.auth.decorators import login_required
from django.shortcuts import render
@login_required
def serverstats_home(request):
return render(request, "serverstats/home.html")
Template home.html
<h2>Server Monitoring Dashboard</h2> <iframe src="https://www.yourdomain.com:42923/d/xxxx?orgId=1&refresh=30s" width="100%" height="900" frameborder="0"> </iframe>
urls.py entry
path("serverstats/", serverstats_home, name="serverstats")
=======================================
11) Security notes
Node exporter is local only ✔
Prometheus is local only ✔
Grafana HTTPS enforced ✔
Random port 42923 ✔
Anonymous Grafana disabled ✔
Django auth required ✔
TLS everywhere ✔
Migrating Web Site From One Vps To Another
update soon
docker basics
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.io
sudo systemctl enable --now docker
Add user to docker group
sudo usermod -aG docker $USER
Test docker
docker run hello-world
Create file:
/etc/docker/daemon.json
Add configuration:
{
"ipv6": false,
"dns": ["8.8.8.8","1.1.1.1"]
}
Restart docker
systemctl restart docker
Verify
docker run hello-world
docker pull nginx
docker pull ubuntu
docker pull mysql
docker run -d --name mynginx -p 8080:80 nginx
Explanation
-d = run in background
--name mynginx = container name
-p 8080:80 = map container port 80 to host port 8080
docker exec -it mynginx bash
docker stop mynginx
docker start mynginx
docker restart mynginx
docker logs mynginx
docker logs -f mynginx
docker stop mynginx
docker rm mynginx
Remove all stopped containers
docker container prune
docker run -d --name mynginx -p 8081:80 nginx
Container port 80 → Host port 8081
Create configuration file
mkdir -p /data/docker
touch /data/docker/my.cnf
chown 999:999 /data/docker/my.cnf
chmod 644 /data/docker/my.cnf
Run MySQL container
docker run -d \
--name mydb \
-e MYSQL_ROOT_PASSWORD=pass123 \
-v /data/mysql:/var/lib/mysql \
-v /data/docker/my.cnf:/etc/mysql/conf.d/my.cnf \
-p 3306:3306 \
mysql:latest
Create Dockerfile
cat > Dockerfile <<EOF
FROM mysql:8.0
LABEL maintainer="you@example.com"
COPY my.cnf /etc/mysql/conf.d/my.cnf
RUN apt-get update && apt-get install -y \
vim \
net-tools \
&& apt-get clean
EOF
docker images # list images
docker ps # list containers
docker stats # container resource usage
docker logs -f name # view logs
docker inspect name # container config
docker exec -it name bash # enter container
docker top name # processes inside container
docker system df # disk usage
docker run -d \
--name mydb5 \
-e MYSQL_ROOT_PASSWORD=pass123 \
-p 3308:3306 \
--memory="1g" \
--cpus="1.5" \
mysql:8.0
Verify limits
docker inspect mydb5 | grep -i -E "memory|cpus"
Live resource usage
docker stats mydb5
Check inside container
docker exec -it mydb5 cat /sys/fs/cgroup/memory/memory.limit_in_bytes
docker exec -it mydb5 cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
docker exec -it mydb5 cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
docker run -d \
--name mydb10 \
--restart=always \
-e MYSQL_ROOT_PASSWORD=pass123 \
-p 3310:3306 \
mysql:latest
Restart options
no
on-failure
always
unless-stopped
Default network: bridge
Create network
docker network create mynet2
docker network inspect mynet2
Run container on network
docker run -d \
--name mydb12 \
--restart=always \
-e MYSQL_ROOT_PASSWORD=pass123 \
-p 3312:3306 \
--network mynet2 \
mysql:latest
Connect container to multiple networks
docker network connect mynet mydb9
docker network connect mynet1 mydb9
docker network connect mynet2 mydb9
docker network connect mynet3 mydb9
Network types
bridge : containers on same host
host : use host network
none : isolated container
overlay : multi-host cluster networking
Example
docker run -d \
--name web2 \
--restart unless-stopped \
--health-cmd="curl -f http://localhost:80 || exit 1" \
--health-interval=30s \
--health-retries=3 \
--health-timeout=5s \
nginx
Exit codes
0 = success
1 = failure
command missing = failure
docker cp temp-tomcat:/usr/local/tomcat/conf /data/mytomcat
docker run -d \
--name mydb16v8 \
-e MYSQL_ROOT_PASSWORD=pass123 \
-e TZ="Asia/Kolkata" \
--restart unless-stopped \
--health-cmd="mysqladmin ping -h localhost -u root --password=pass123 || exit 1" \
--health-interval=5s \
--health-retries=5 \
-v /data/mysql16v8:/var/lib/mysql \
-v /data/docker/my.cnf:/etc/mysql/conf.d/my.cnf \
-p 3316:3306 \
--memory="1g" \
--cpus="1.5" \
--network mynet2 \
my-mysql-image:8.0
docker run -d \
--name tomcat3 \
--restart unless-stopped \
--health-cmd="curl -f http://localhost:8080 || exit 1" \
--health-interval=30s \
--health-retries=3 \
--health-timeout=5s \
-e TZ="Asia/Kolkata" \
-e JAVA_OPTS="-Xms128m -Xmx256m -Duser.timezone=Asia/Kolkata" \
-p 8083:8080 \
--network mynet2 \
-v /data/tomcat_docker/tomcat_common/warfile/log-api-1.0.war:/usr/local/tomcat/webapps/log-api-1.0.war \
-v /data/tomcat_docker/tomcat3/logs:/usr/local/tomcat/logs \
-v /data/tomcat_docker/tomcat_common/db_properties/db.properties:/usr/local/tomcat/conf/db.properties \
tomcat:9.0.111-jdk8-corretto-al2
docker run -d \
--name nginxlb \
--restart unless-stopped \
--network mynet2 \
-p 80:80 \
-p 443:443 \
-v /data/nginx-lb/nginx.conf:/etc/nginx/nginx.conf:ro \
-v /data/nginx-lb/SSL:/etc/nginx/SSL:ro \
-v /data/nginx-lb/logs:/var/log/nginx \
--health-cmd="sh -c 'echo > /dev/tcp/127.0.0.1/80 || exit 1'" \
--health-interval=30s \
--health-retries=3 \
--health-timeout=5s \
-e TZ="Asia/Kolkata" \
nginx:stable
Test request
curl -k "https://localhost:443/log-api-1.0/log?msgtext=thisistest4011116&status=ok"
Start containers
docker compose up -d
Stop containers
docker compose down
Rebuild containers
docker compose up -d --build
Force recreate
docker compose up -d --force-recreate
==================== OFFLINE KUBERNETES INSTALL (RHEL 8.10) ====================
CLUSTER DETAILS
---------------
MASTER : 192.168.241.160
WORKERS : 192.168.241.161 , 192.168.241.162
K8S : v1.30.14
RUNTIME : containerd
CNI : flannel
ARTIFACTS PATH : /data
packages in /data: conntrack-tools-1.4.4-11.el8.x86_64, containerd.io-1.6.32-3.1.el8.x86_64, cri-tools-1.30.1-150500.1.1.x86_64, ethtool-5.13-2.el8.x86_64, iproute-6.2.0-6.el8_10.x86_64, iproute-tc-6.2.0-6.el8_10.x86_64, iptables-1.8.5-11.el8_9.x86_64, iptables-ebtables-1.8.5-11.el8_9.x86_64, kubeadm-1.30.14-150500.1.1.x86_64, kubectl-1.30.14-150500.1.1.x86_64, kubelet-1.30.14-150500.1.1.x86_64, kubernetes-cni-1.4.0-150500.1.1.x86_64, socat-1.7.4.1-2.el8_10.x86_64, createrepo, bash-auoconnect
================================================================================
STEP 0 : COMMON SETUP (RUN ON ALL NODES)
================================================================================
swapoff -a
sed -i '/swap/d' /etc/fstab
cat <<EOF >/etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat <<EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
EOF
sysctl --system
================================================================================
STEP 1 : CONFIGURE OFFLINE REPO (RUN ON ALL NODES)
================================================================================
cat <<EOF >/etc/yum.repos.d/k8s-offline.repo
[k8s-offline]
name=Kubernetes Offline Repo
baseurl=file:///data/k8s-rpms
enabled=1
gpgcheck=0
EOF
dnf clean all
================================================================================
STEP 2 : INSTALL PACKAGES (RUN ON ALL NODES)
================================================================================
dnf install -y \
containerd.io \
kubeadm kubelet kubectl cri-tools kubernetes-cni \
conntrack-tools iproute iproute-tc iptables iptables-ebtables ethtool socat
systemctl enable --now containerd
systemctl enable kubelet
================================================================================
STEP 3 : CONFIGURE CONTAINERD (RUN ON ALL NODES)
================================================================================
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' \
/etc/containerd/config.toml
systemctl restart containerd
systemctl status containerd
================================================================================
STEP 4 : IMPORT IMAGES (OFFLINE)
================================================================================
# MASTER ONLY
ctr -n k8s.io images import /data/offline/k8s-images.tar
ctr -n k8s.io images import /data/offline/flannel.tar
# WORKERS ONLY
ctr -n k8s.io images import /data/offline/k8s-images.tar
================================================================================
STEP 5 : INITIALIZE CLUSTER (MASTER ONLY)
================================================================================
kubeadm init \
--apiserver-advertise-address=192.168.241.160 \
--pod-network-cidr=10.244.0.0/16
================================================================================
STEP 6 : CONFIGURE kubectl (MASTER ONLY)
================================================================================
mkdir -p $HOME/.kube
cp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
================================================================================
STEP 7 : INSTALL FLANNEL (MASTER ONLY, OFFLINE)
================================================================================
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl apply -f /data/offline/kube-flannel.yml
kubectl get pods -n kube-system
================================================================================
STEP 8 : JOIN WORKER NODES
================================================================================
# ON MASTER
kubeadm token create --print-join-command
# RUN OUTPUT COMMAND ON EACH WORKER
kubeadm reset -f
rm -rf /etc/cni/net.d
rm -rf /var/lib/cni
rm -rf /var/lib/kubelet/*
systemctl restart containerd
systemctl restart kubelet
kubeadm join 192.168.241.160:6443 \
--token <TOKEN> \
--discovery-token-ca-cert-hash sha256:<HASH>
================================================================================
STEP 9 : VERIFY CLUSTER (MASTER ONLY)
================================================================================
kubectl get nodes -o wide
EXPECTED OUTPUT
---------------
control Ready control-plane
node1 Ready
node2 Ready
==================== OFFLINE KUBERNETES INSTALL COMPLETE =======================
####################### KUBERNETES ETCD FULL LAB (BACKUP + BREAK + RESTORE) #######################
################################ STEP 1: CHECK CLUSTER ############################################
kubectl get nodes
kubectl get pods -A
##############################################################################################
################################ STEP 2: TAKE BACKUP (NO etcdctl ON HOST) ####################
exec -n kube-system etcd-controlnode -- etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --endpoints=https://127.0.0.1:2379 snapshot save /var/lib/etcd-backup3.db
##############################################################################################
################################ STEP 3: BREAK CLUSTER ########################################
kubectl delete deployment webserver-deployment
kubectl delete pod samplepod
kubectl get all
##############################################################################################
################################ STEP 4: STOP ETCD ############################################
mv /etc/kubernetes/manifests/etcd.yaml /tmp/
# Verify
crictl ps | grep etcd
##############################################################################################
################################ STEP 5: RESTORE SNAPSHOT #####################################
ctr -n k8s.io run --rm -t
--mount type=bind,src=/etc/kubernetes/pki/etcd,dst=/etc/kubernetes/pki/etcd,options=rbind:rw
--mount type=bind,src=/var/lib,dst=/var/lib,options=rbind:rw
registry.k8s.io/etcd:3.5.16-0 etcd-restore sh
# Inside container:
ETCDCTL_API=3 etcdctl snapshot restore /var/lib/etcd/etcd-backup.db
--data-dir=/var/lib/etcd-restore
# VERIFY (VERY IMPORTANT)
echo /var/lib/*
exit
##############################################################################################
################################ STEP 6: UPDATE ETCD MANIFEST #################################
vi /etc/kubernetes/manifests/etcd.yaml
# CHANGE THESE THREE PLACES:
--data-dir=/var/lib/etcd-restore
# volumeMounts:
* mountPath: /var/lib/etcd-restore
# volumes:
path: /var/lib/etcd-restore
##############################################################################################
################################ STEP 7: START ETCD ###########################################
mv /tmp/etcd.yaml /etc/kubernetes/manifests/
# Wait 20–30 seconds
##############################################################################################
################################ STEP 8: FIX AUTH (VERY IMPORTANT) ############################
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sleep 5
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/
mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/
sleep 5
mv /tmp/kube-controller-manager.yaml /etc/kubernetes/manifests/
mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
sleep 5
mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/
##############################################################################################
################################ STEP 9: (OPTIONAL) BYPASS AUTH ###############################
vi /etc/kubernetes/manifests/kube-apiserver.yaml
# Add:
--authorization-mode=AlwaysAllow
##############################################################################################
################################ STEP 10: VERIFY ##############################################
crictl ps
ps -ef | grep etcd
kubectl get nodes
kubectl get pods -A
##############################################################################################
################################ FINAL MEMORY #################################################
Backup → Break → Stop → Restore → Fix Mount → Start → Restart Control Plane → Verify
##############################################################################################
calico-kube-controllers pod was stuck in CrashLoopBackOff and pods could not communicate with the Kubernetes API server via ClusterIP 10.96.0.1:443.
| Component | Detail |
|---|---|
| OS | RHEL 9.6 |
| Kubernetes | v1.29.15 |
| CNI | Calico v3.27.0 |
| Container Runtime | containerd 2.2.2 |
| Node IPs | 192.168.241.140/141/142 |
| Pod CIDR (configured) | 192.168.0.0/16 |
| Service CIDR | 10.96.0.0/12 |
calico-kube-controllers pod stuck in CrashLoopBackOffdial tcp 10.96.0.1:443: i/o timeout10.96.0.110.96.0.1:443 directly from the host (got 403 Forbidden — meaning host-level connectivity was fine)net.ipv4.ip_forward = 1) on all nodes ✅10.96.0.1 existed on all nodes ✅10.96.0.1:443 directly — working on all nodes ✅Host-to-ClusterIP worked but pod-to-ClusterIP timed out. This pointed to a problem specifically with how pod traffic was being NAT'd through the ClusterIP rules.
Running this command on workernode1:
bash
iptables -t nat -L KUBE-SVC-NPX46M4PTMTKRN6Y -v -n
```
Revealed this rule:
```
KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.0.1 tcp dpt:443
```
The `!192.168.0.0/16` means — **only masquerade (SNAT) traffic coming from OUTSIDE 192.168.0.0/16**. Traffic from inside that range is excluded from masquerading.
---
## Root Cause
**Pod CIDR `192.168.0.0/16` overlapped with Node IP range `192.168.241.x`.**
This caused a chain reaction:
```
Pod IP: 192.168.212.4
↓
Sends packet to 10.96.0.1:443
↓
kube-proxy KUBE-SERVICES chain matches → forwards to KUBE-SVC-NPX46M4PTMTKRN6Y
↓
KUBE-MARK-MASQ rule checks source IP:
192.168.212.4 is INSIDE 192.168.0.0/16
↓
MASQUERADE is SKIPPED ← problem here
↓
Packet reaches API server (192.168.241.140:6443)
with source IP 192.168.212.4 (pod IP)
↓
API server tries to reply to 192.168.212.4
but has no route back to that pod IP
↓
Connection times out
kube-proxy intentionally excludes pod CIDR from masquerading to avoid unnecessary NAT for pod-to-pod traffic. But when the pod CIDR overlaps with the node network, this optimization breaks pod-to-ClusterIP communication.
| Source | Source IP | In 192.168.0.0/16? | Masqueraded? | Works? |
|---|---|---|---|---|
| Node (host) | 192.168.241.x | Yes | No | ✅ Yes — node IP is routable |
| Pod | 192.168.212.4 | Yes | No | ❌ No — pod IP not directly routable to API server |
Nodes have real routable IPs so replies come back fine even without masquerading. Pods do not — they need SNAT so the reply goes back to the node, which then forwards it to the pod.
Set masqueradeAll: true in kube-proxy configmap:
yaml
iptables:
masqueradeAll: true
This forces SNAT on all pod-to-ClusterIP traffic regardless of source IP, bypassing the overlap problem. This worked but adds NAT overhead on every pod connection.
| Network | Old (broken) | New (correct) |
|---|---|---|
| Pod CIDR | 192.168.0.0/16 |
172.16.0.0/16 |
| Service CIDR | 10.96.0.0/12 |
10.96.0.0/12 |
| Node IPs | 192.168.241.x |
192.168.241.x |
Reinstall command:
bash
kubeadm init \
--pod-network-cidr=172.16.0.0/16 \
--service-cidr=10.96.0.0/12 \
--apiserver-advertise-address=192.168.241.140
With Calico configured to match:
yaml
- name: CALICO_IPV4POOL_CIDR
value: "172.16.0.0/16"
192.168.0.0/16 pod CIDR is just a default, not a requirement — it can and should be changed if your node network uses the same range.Ssl Full Setup (Ca + Server Cert + Verify + Test)
openssl genrsa -out abhilash-ca.key 3072
openssl req -x509 -new -nodes
-key abhilash-ca.key
-sha256 -days 3650
-out abhilash-ca.crt
-subj "/C=IN/ST=Maharashtra/L=Mumbai/O=AbhilashOrg/CN=Abhilash-Root-CA"
openssl genrsa -out server.key 3072
cat < san.cnf
[req]
distinguished_name = dn
req_extensions = req_ext
prompt = no
[dn]
C = IN
ST = Maharashtra
L = Mumbai
O = AbhilashOrg
CN = nginx.local
[req_ext]
subjectAltName = @alt_names
[alt_names]
DNS.1 = nginx.local
DNS.2 = controlnode
IP.1 = 127.0.0.1
IP.2 = 192.168.240.140
EOF
openssl req -new
-key server.key
-out server.csr
-config san.cnf
openssl x509 -req
-in server.csr
-CA abhilash-ca.crt
-CAkey abhilash-ca.key
-CAcreateserial
-out server.crt
-days 825
-sha256
-extensions req_ext
-extfile san.cnf
openssl verify -CAfile abhilash-ca.crt server.crt
openssl x509 -in server.crt -text -noout | grep -A1 "Subject Alternative Name"
openssl x509 -noout -modulus -in server.crt | openssl md5
openssl rsa -noout -modulus -in server.key | openssl md5
openssl s_server -key server.key -cert server.crt -accept 8443
kubectl create secret tls nginx-tls
--cert=server.crt
--key=server.key
🔥 Volume Types Summary
Type Persistent Multi-node Use Case
emptyDir ❌ ❌ temp
hostPath ❌ ❌ node access
PV/PVC ✅ depends DB
NFS ✅ ✅ shared
StorageClass ✅ depends auto
ConfigMap ❌ ❌ config
Secret ❌ ❌ sensitive data
persistentVolumeClaimPolicy
persitentVolumeReclaimPolicy
persistentVolumeReclaimPolicy
=============================
persistenVolumeReclaimPolicy
persistantVolumeReclaimPolicy
pv
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv-secure
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany # NFS supports multiple pods
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys # default auth (explicit)
- nfsvers=4.1 # stable version
- hard # safe retry behavior
nfs:
server: 192.168.1.10 # 👈 your NFS server IP
path: /data/nfsshared/03042026
readOnly: false
Common Units You Should Know
Unit Meaning
Ki Kilobyte
Mi Megabyte
Gi Gigabyte
Ti Terabyte
-------------------------------------------------------------------------
| Mode | Meaning |
| ------------------- | ----------------------- |
| ReadWriteOnce (RWO) | One node only |
| ReadOnlyMany (ROX) | Many pods read only |
| ReadWriteMany (RWX) | Many pods read/write |
==================================================================================
persistentVolumeReclaimPolicy: Retain
| Policy | Behavior |
| -------------------- | --------------- |
| Retain | Keep data |
| Delete | Delete storage |
| Recycle (deprecated) | Clean and reuse |
==============================================================================================
PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-combined
spec:
volumeName: nfs-pv-1 # 👈 exact PV
accessModes:
- ReadWriteMany
storageClassName: "" # 👈 ignore any StorageClass
resources:
requests:
storage: 1Gi
selector:
matchLabels:
app: app1 # 👈 must match PV label
apiVersion: v1
kind: PersistentVolume
metadata:
name: pvc4
labels:
type: pvc4
spec:
volumeName: pv4
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 1Gi
selector:
matchLabels:
type: pv4
=========================================================================================================
POD with Volumes
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: sample4
name: sample4-vol-pod
spec:
containers:
- image: nginx:1.25
name: sample4-vol-pod
ports:
- containerPort: 80
command: ["/bin/sh","-c"]
args: ["while true; do echo \"this is test $(date)\" > /data/file.txt; sleep 1; done;"]
env:
- name: TZ
value: IST-5:30
envFrom:
- configMapRef:
name: configmap4
- secretRef:
name: secret4
volumeMounts:
- name: sample4-vol-pod-pvc
mountPath: /app
- name: sample4-vol-pod-emptydir
mountPath: /emptydir
- name: sample4-vol-pod-hostonly
mountPath: /hostpath
- name: configmap4-vol
mountPath: /etc/config
- name: secret5-vol
mountPath: /etc/secret5
volumes:
- name: sample4-vol-pod-pvc
persistentVolumeClaim:
claimName: pvc4
- name: sample4-vol-pod-emptydir
emptyDir: {}
- name: sample4-vol-pod-hostonly
hostPath:
path: /data/k8/hostpath-vol-smaple4
type: DirectoryOrCreate
- name: configmap4-vol
configMap:
name: configmap4
- name: secret5-vol
secret:
secretName: secret5
======================================================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv4
labels:
type: pv4
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: 192.168.241.140
path: /data/nfsshared/date-04march
readOnly: false
====================================================================================
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc4
labels:
type: pvc4
spec:
volumeName: pv4
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 1Gi
selector:
matchLabels:
type: pv4
========================================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap4
labels:
type: configmap4
data:
APP_COLOR: blue
APP_MODE: dev
=======================================================================================
apiVersion: v1
kind: Secret
metadata:
name: secret5
labels:
type: secret5
data:
USERNAME: YWRtaW4K
PASSWORD: YWRtaW4K
=========================================
kubectl create secret generic my-secret \
--from-literal=key=value
| Type | Purpose |
| --------------- | ------------------ |
| generic | Any key-value |
| tls | HTTPS certs |
| docker-registry | Private image pull |
| basic-auth | username/password |
| ssh-auth | SSH keys |
=======================================================================
kubectl create configmap my-config-dir \
--from-file=/path/to/config-dir/
kubectl create configmap my-config \
--from-literal=APP_COLOR=blue \
--from-literal=APP_MODE=dev
===============================
openssl genrsa -out abhilash-ca.key 3072
# 🚀 RabbitMQ Cluster on Kubernetes (Complete Setup + Troubleshooting Guide)
---
# 📌 Objective
Deploy a **3-node RabbitMQ Cluster** on Kubernetes with:
* High Availability
* Persistent Storage (NFS)
* Auto Clustering
* Management UI
* Application connectivity (Tomcat)
---
# 🏗️ Components Created
## 1. Persistent Volumes (NFS)
We created 3 PVs:
* pv-rabbitmq1
* pv-rabbitmq2
* pv-rabbitmq3
Each mapped to:
```text
/data/nfsshared/rabbitmq-pv1
/data/nfsshared/rabbitmq-pv2
/data/nfsshared/rabbitmq-pv3
```
Used:
```yaml
accessModes: ReadWriteOnce
```
👉 Ensures **1 pod = 1 storage**
---
## 2. ConfigMap
Contains:
### enabled_plugins
```erlang
[rabbitmq_management,rabbitmq_peer_discovery_k8s].
```
### rabbitmq.conf
```ini
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = service-rabbitmq-headless
cluster_formation.k8s.hostname_suffix = .service-rabbitmq-headless.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
queue_master_locator=min-masters
```
👉 Enables **auto clustering using Kubernetes**
---
## 3. RBAC (CRITICAL)
```yaml
ServiceAccount → rabbitmq
Role → access pods, endpoints
RoleBinding → bind both
```
👉 Required because:
```text
RabbitMQ calls Kubernetes API → needs permission
```
---
## 4. Headless Service
```yaml
name: service-rabbitmq-headless
clusterIP: None
publishNotReadyAddresses: true
```
👉 Enables DNS like:
```text
rabbitmq-0.service-rabbitmq-headless
```
---
## 5. NodePort Service (UI)
```yaml
port: 15672
nodePort: 30072
```
👉 Access UI:
```text
http://<NodeIP>:30072
```
---
## 6. ClusterIP Service (App)
```yaml
name: rabbitmq-svc
port: 5672
```
👉 Used by:
```text
Tomcat → rabbitmq-svc:5672
```
---
## 7. StatefulSet
Key points:
```yaml
serviceName: service-rabbitmq-headless
replicas: 3
```
### ENV:
```yaml
RABBITMQ_DEFAULT_USER=admin
RABBITMQ_DEFAULT_PASS=admin
RABBITMQ_ERLANG_COOKIE=mysecretcookie
RABBITMQ_USE_LONGNAME=true
```
### Volumes:
* PVC → /var/lib/rabbitmq
* ConfigMap → rabbitmq.conf + plugins
👉 Ensures:
* Stable identity
* Persistent data
* Config-driven clustering
---
# ⚙️ FINAL EXECUTION ORDER (VERY IMPORTANT)
👉 Always follow this order:
```bash
kubectl apply -f pv-rabbit-01.yaml
kubectl apply -f pv-rabbit-02.yaml
kubectl apply -f pv-rabbit-03.yaml
kubectl apply -f rbac-rabbitmq.yaml
kubectl apply -f configmap-rabbit.yaml
kubectl apply -f service-rabbitmq-headless.yaml
kubectl apply -f service-rabbitmq-svc.yaml
kubectl apply -f service-rabbitmq-nodeport.yaml
kubectl apply -f StatefulSet-rabbitmq.yaml
```
---
# 🔥 TROUBLESHOOTING JOURNEY
---
## ❌ Issue 1: DNS Not Working
Problem:
```text
rabbitmq-1 not resolving
```
Fix:
```yaml
publishNotReadyAddresses: true
```
---
## ❌ Issue 2: Service Name Mismatch
Problem:
```text
rabbitmq-headless vs service-rabbitmq-headless
```
Fix:
```text
Must match EXACTLY
```
---
## ❌ Issue 3: No rabbitmq.conf
Fix:
Added clustering config
---
## ❌ Issue 4: 403 Error (CRITICAL)
Log:
```text
Failed to fetch nodes from Kubernetes API: 403
```
Fix:
Added RBAC
---
## ❌ Issue 5: Short vs Long Names
Error:
```text
epmd nxdomain
```
Fix:
```yaml
RABBITMQ_USE_LONGNAME=true
```
---
## ❌ Issue 6: Cluster Join Failure
Error:
```text
tables_not_present
mnesia_not_running
```
👉 Root cause:
```text
Pods not ready at same time (timing issue)
```
---
## ❌ Issue 7: Cluster Not Forming
Final log:
```text
Starting as a blank standalone node
```
👉 Reason:
```text
Retry failed → node becomes standalone
```
---
# 🧠 WHY THIS HAPPENS
RabbitMQ:
```text
Cluster formation happens ONLY at startup
```
If peers not ready → join fails
---
# 🔧 FINAL FIXES APPLIED
* Enabled RBAC ✅
* Enabled longnames ✅
* Fixed serviceName ✅
* Fixed DNS ✅
* Added retry logic ✅
* Restarted pods cleanly ✅
---
# 📊 FINAL RESULT
```bash
rabbitmqctl cluster_status
```
Output:
```text
rabbit@rabbitmq-0
rabbit@rabbitmq-1
rabbit@rabbitmq-2
```
---
# 🎯 WHAT WE ACHIEVED
✅ 3-node RabbitMQ cluster
✅ Auto discovery via Kubernetes
✅ Persistent storage
✅ UI access
✅ App connectivity
✅ HA-ready setup
---
# ⚠️ ALTERNATIVES
| Approach | Result |
| -------------- | ------------------------ |
| No RBAC | No clustering ❌ |
| Manual join | Works but not stable ⚠️ |
| Classic config | Static, not scalable ❌ |
| Helm chart | Best production option ✅ |
---
# 🧠 FINAL LEARNING
* Kubernetes = dynamic → needs API
* RabbitMQ = startup-based clustering
* RBAC = mandatory
* Headless service = must
* Longnames = required
* Timing = critical
---
# 🚀 NEXT STEPS
* Create quorum queues
* Test failover (kill pod)
* Connect Tomcat producer/consumer
* Monitor cluster
---
# 📌 FINAL CONCLUSION
You successfully built a **production-grade RabbitMQ cluster on Kubernetes**
and solved real-world issues like:
* DNS
* RBAC
* Clustering
* Node naming
* Startup timing
---
==================================configmap-rabbit.yaml========================================
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap-rabbit
labels:
type: configmap-rabbit
data:
enabled_plugins: |
[rabbitmq_management,rabbitmq_peer_discovery_k8s].
rabbitmq.conf: |
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = service-rabbitmq-headless
cluster_formation.k8s.hostname_suffix = .service-rabbitmq-headless.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
queue_master_locator=min-masters
==================================pv-rabbit-01.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq1
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv1
readOnly: false
==================================pv-rabbit-02.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq2
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv2
readOnly: false
==================================pv-rabbit-03.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq3
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv3
readOnly: false
==================================rbac-rabbitmq.yaml========================================
apiVersion: v1
kind: ServiceAccount
metadata:
name: rabbitmq
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rabbitmq
namespace: default
rules:
- apiGroups: [""]
resources:
- endpoints
- pods
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rabbitmq
namespace: default
subjects:
- kind: ServiceAccount
name: rabbitmq
namespace: default
roleRef:
kind: Role
name: rabbitmq
apiGroup: rbac.authorization.k8s.io
==================================service-rabbitmq-headless.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: service-rabbitmq-headless
labels:
type: service-rabbitmq-headless
spec:
clusterIP: None
publishNotReadyAddresses: true
selector:
app: rabbitmq
ports:
- name: amqp
port: 5672
- name: management
port: 15672
- name: epmd
port: 4369
- name: cluster-rpc
port: 25672
==================================service-rabbitmq-nodeport.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-nodeport
labels:
type: rabbitmq-nodeport
spec:
type: NodePort
selector:
app: rabbitmq
ports:
- name: management
port: 15672
targetPort: 15672
nodePort: 30072
==================================service-rabbitmq-svc.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-svc
labels:
type: rabbitmq-svc
spec:
type: ClusterIP
selector:
app: rabbitmq
ports:
- name: amqp
port: 5672
targetPort: 5672
==================================StatefulSet-rabbitmq.yaml========================================
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rabbitmq
labels:
type: rabbitmq
spec:
serviceName: service-rabbitmq-headless
replicas: 3
selector:
matchLabels:
app: rabbitmq
template:
metadata:
labels:
app: rabbitmq
spec:
serviceAccountName: rabbitmq
containers:
- name: rabbitmq
image: rabbitmq:3.12-management
ports:
- containerPort: 5672
- containerPort: 15672
env:
- name: RABBITMQ_DEFAULT_USER
value: "admin"
- name: RABBITMQ_DEFAULT_PASS
value: "admin"
- name: RABBITMQ_ERLANG_COOKIE
value: "mysecretcookie"
- name: RABBITMQ_USE_LONGNAME
value: "true"
volumeMounts:
- name: data
mountPath: /var/lib/rabbitmq
- name: config
mountPath: /etc/rabbitmq/enabled_plugins
subPath: enabled_plugins
- name: rabbitconf
mountPath: /etc/rabbitmq/rabbitmq.conf
subPath: rabbitmq.conf
volumes:
- name: config
configMap:
name: configmap-rabbit
- name: rabbitconf
configMap:
name: configmap-rabbit
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: ""
resources:
requests:
storage: 1Gi
selector:
matchLabels:
type: pv-rabbitmq
# 👍 END
| Task | Imperative Command | Declarative YAML |
| Create Namespace | kubectl create namespace dev | apiVersion: v1 kind: Namespace metadata: name: dev |
| Create Pod | kubectl run nginx-pod --image=nginx:1.25 | apiVersion: v1 kind: Pod metadata: name: nginx-pod spec: containers: - name: nginx image: nginx:1.25 |
| Create Deployment | kubectl create deployment sampledeploy --image=nginx:1.25 --replicas=4 | apiVersion: apps/v1 kind: Deployment metadata: name: sampledeploy spec: replicas: 4 selector: matchLabels: app: sampledeploy template: metadata: labels: app: sampledeploy spec: containers: - name: nginx image: nginx:1.25 |
| Create ClusterIP Service | kubectl expose deployment sampledeploy --port=80 --target-port=80 | apiVersion: v1 kind: Service metadata: name: sampledeploy-service spec: selector: app: sampledeploy ports: - port: 80 targetPort: 80 |
| Create NodePort Service | kubectl expose deployment sampledeploy --type=NodePort --port=80 --target-port=80 | apiVersion: v1 kind: Service metadata: name: sampledeploy-nodeport spec: type: NodePort selector: app: sampledeploy ports: - port: 80 targetPort: 80 nodePort: 30080 |
| Create ConfigMap | kubectl create configmap app-config --from-literal=APP_MODE=production --from-literal=COLOR=blue | apiVersion: v1 kind: ConfigMap metadata: name: app-config data: APP_MODE: production COLOR: blue |
| Create Secret | kubectl create secret generic db-secret --from-literal=username=admin --from-literal=password=pass123 | apiVersion: v1 kind: Secret metadata: name: db-secret type: Opaque stringData: username: admin password: pass123 |
| Create ServiceAccount | kubectl create serviceaccount frontend-sa | apiVersion: v1 kind: ServiceAccount metadata: name: frontend-sa |
| Create Role | kubectl create role pod-reader --verb=get,list,watch --resource=pods | apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] |
| Create RoleBinding | kubectl create rolebinding pod-reader-binding --role=pod-reader --serviceaccount=default:frontend-sa | apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pod-reader-binding subjects: - kind: ServiceAccount name: frontend-sa namespace: default roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io |
| Create ClusterRole | kubectl create clusterrole node-reader --verb=get,list,watch --resource=nodes | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: node-reader rules: - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] |
| Create ClusterRoleBinding | kubectl create clusterrolebinding node-reader-binding --clusterrole=node-reader --serviceaccount=default:frontend-sa | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-reader-binding subjects: - kind: ServiceAccount name: frontend-sa namespace: default roleRef: kind: ClusterRole name: node-reader apiGroup: rbac.authorization.k8s.io |
| Create PVC | Usually declarative | apiVersion: v1 kind: PersistentVolumeClaim metadata: name: app-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi |
| Create StatefulSet | Usually declarative | apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb spec: serviceName: mongodb-headless replicas: 1 selector: matchLabels: app: mongodb template: metadata: labels: app: mongodb spec: containers: - name: mongodb image: mongo:6 |
| Create Job | kubectl create job test-job --image=busybox | apiVersion: batch/v1 kind: Job metadata: name: test-job spec: template: spec: containers: - name: busybox image: busybox restartPolicy: Never |
| Create CronJob | kubectl create cronjob backup-job --image=busybox --schedule="*/5 * * * *" | apiVersion: batch/v1 kind: CronJob metadata: name: backup-job spec: schedule: "*/5 * * * *" jobTemplate: spec: template: spec: containers: - name: busybox image: busybox restartPolicy: Never |
###############################
# MYSQL 8.0.x → 8.4.x UPGRADE
# CLEAN INSTALL + RESTORE METHOD
###############################
ENVIRONMENT:
- RHEL8 / Rocky Linux
- Custom datadir: /data/mysql_server
- No GTID
- Old MySQL 8.0.x
- New MySQL 8.4.x
- Using encryption/keyring
- Fresh initialize method
====================================================
STEP 1 — CHECK OLD MYSQL ENVIRONMENT
====================================================
mysql -uroot -p
SHOW PLUGINS;
SELECT TABLE_SCHEMA,TABLE_NAME,CREATE_OPTIONS
FROM information_schema.tables
WHERE CREATE_OPTIONS LIKE '%ENCRYPTION%';
grep -Ri keyring /etc/my.cnf*
====================================================
STEP 2 — TAKE FULL BACKUP
====================================================
mysqldump \
--all-databases \
--routines \
--events \
--triggers \
--single-transaction \
--hex-blob \
-u root -p > /backup/full.sql
====================================================
STEP 3 — BACKUP KEYRING (VERY IMPORTANT)
====================================================
tar -cvzf /backup/mysql-keyring.tar.gz \
/data/mysql-keyring
====================================================
STEP 4 — BACKUP CONFIG
====================================================
cp -p /etc/my.cnf /backup/
====================================================
STEP 5 — REMOVE ENCRYPTION FROM DUMP
(RECOMMENDED SAFEST METHOD)
====================================================
cp /backup/full.sql /backup/full_no_encrypt.sql
sed -i "s/ENCRYPTION='Y'/ENCRYPTION='N'/g" \
/backup/full_no_encrypt.sql
====================================================
STEP 6 — STOP MYSQL
====================================================
systemctl stop mysqld
ps -ef | grep mysqld
IF STILL RUNNING:
pkill -9 mysqld
====================================================
STEP 7 — REMOVE OLD MYSQL RPMs
====================================================
rpm -qa | grep -i mysql
dnf remove mysql*
OR
rpm -e mysql-community-server \
mysql-community-client \
mysql-community-common \
mysql-community-libs
====================================================
STEP 8 — INSTALL MYSQL 8.4 RPMs
====================================================
cd /data/pkg/mysql84/
yum install mysql-community-*.rpm
OR
rpm -ivh mysql-community-common-8.4*.rpm
rpm -ivh mysql-community-client-plugins-8.4*.rpm
rpm -ivh mysql-community-libs-8.4*.rpm
rpm -ivh mysql-community-client-8.4*.rpm
rpm -ivh mysql-community-server-8.4*.rpm
====================================================
STEP 9 — RENAME OLD DATADIR
====================================================
mv /data/mysql_server \
/data/mysql_server_80_backup
====================================================
STEP 10 — CREATE NEW DATADIR
====================================================
mkdir -p /data/mysql_server
chown -R mysql:mysql /data/mysql_server
chmod 750 /data/mysql_server
====================================================
STEP 11 — EDIT /etc/my.cnf
====================================================
REMOVE OLD KEYRING CONFIG:
#early-plugin-load=keyring_file.so
#keyring_file_data=/data/mysql-keyring/keyring
====================================================
STEP 12 — CREATE NEW KEYRING DIRECTORY
====================================================
mkdir -p /data/mysql-keyring
chown -R mysql:mysql /data/mysql-keyring
chmod 750 /data/mysql-keyring
====================================================
STEP 13 — CREATE COMPONENT CONFIG
====================================================
mkdir -p /var/lib/mysql-files
vi /var/lib/mysql-files/component_keyring_file.cnf
ADD:
{
"path": "/data/mysql-keyring/keyring",
"read_only": false
}
SAVE FILE
====================================================
STEP 14 — FIX PERMISSIONS
====================================================
chown mysql:mysql \
/var/lib/mysql-files/component_keyring_file.cnf
chmod 640 \
/var/lib/mysql-files/component_keyring_file.cnf
====================================================
STEP 15 — CREATE BOOTSTRAP FILE
(VERY IMPORTANT)
====================================================
vi /usr/sbin/mysqld.my
ADD:
{
"components": "file://component_keyring_file"
}
SAVE FILE
====================================================
STEP 16 — FIX BOOTSTRAP FILE PERMISSIONS
====================================================
chown mysql:mysql /usr/sbin/mysqld.my
chmod 640 /usr/sbin/mysqld.my
====================================================
STEP 17 — INITIALIZE MYSQL 8.4
====================================================
mysqld \
--defaults-file=/etc/my.cnf \
--initialize \
--user=mysql
====================================================
STEP 18 — START MYSQL
====================================================
systemctl start mysqld
====================================================
STEP 19 — CHECK LOGS
====================================================
journalctl -xeu mysqld
tail -f /data/mysql_server/mysqld.log
====================================================
STEP 20 — LOGIN MYSQL
====================================================
mysql -uroot -p
====================================================
STEP 21 — VERIFY KEYRING COMPONENT
====================================================
SELECT * FROM performance_schema.keyring_component_status;
IF NOT EMPTY = SUCCESS
====================================================
STEP 22 — REGISTER COMPONENT
====================================================
INSTALL COMPONENT 'file://component_keyring_file';
SELECT * FROM mysql.component;
====================================================
STEP 23 — TEST ENCRYPTION
====================================================
CREATE DATABASE testdb;
USE testdb;
CREATE TABLE t1 (
id INT
) ENCRYPTION='Y';
====================================================
STEP 24 — RESTORE DUMP
====================================================
mysql -uroot -p < /backup/full_no_encrypt.sql
====================================================
STEP 25 — VERIFY DATABASES
====================================================
SHOW DATABASES;
SELECT user,host FROM mysql.user;
====================================================
STEP 26 — OPTIONAL RE-ENABLE ENCRYPTION LATER
====================================================
ALTER TABLE table_name ENCRYPTION='Y';
====================================================
IMPORTANT NOTES
====================================================
1. NEVER DELETE:
/data/mysql-keyring
2. NEVER MIX:
old plugin + new component
3. DO NOT USE:
early-plugin-load=keyring_file.so
4. NEW MYSQL 8.4 USES:
component_keyring_file
5. MOST IMPORTANT FILE:
/usr/sbin/mysqld.my
WITHOUT IT:
- component installs
- but encryption fails
6. IF RESTORE FAILS:
- use ENCRYPTION='N'
- restore first
- re-enable later
###############################
END
###############################