KUBERNETES SECURITYCONTEXT — CKA REVISION
1. WHAT IS SECURITYCONTEXT?
---------------------------
SecurityContext defines security-related settings for a Pod or Container.
It controls:
- Linux user/group
- Root vs non-root
- File permissions
- Linux capabilities
- Privilege escalation
- Privileged mode
- Read-only filesystem
- Seccomp
- SELinux
Two levels:
- Pod-level securityContext → applies to Pod/containers by default
- Container-level securityContext → applies specifically to that container
If a setting exists at both Pod and Container level, container-level setting takes precedence.
2. POD-LEVEL SECURITYCONTEXT
----------------------------
Example:
spec:
securityContext:
runAsUser: 1000
runAsGroup: 3000
runAsNonRoot: true
fsGroup: 2000
3. runAsUser
------------
Specifies the Linux UID that the container process runs as.
securityContext:
runAsUser: 1000
Meaning:
- Process runs as UID 1000
- Instead of root (UID 0)
Purpose:
- Avoid running applications as root
- Improve security
4. runAsGroup
-------------
Specifies the primary Linux GID.
securityContext:
runAsUser: 1000
runAsGroup: 3000
Meaning:
- UID = 1000
- Primary GID = 3000
5. runAsNonRoot
---------------
Ensures the container does NOT run as root.
securityContext:
runAsNonRoot: true
Important:
runAsNonRoot: true does NOT mean UID 1000.
It means:
"The process must not run as UID 0."
Difference:
runAsUser → "Run as this UID"
runAsNonRoot → "Do NOT run as root"
6. fsGroup
----------
Defines the group ownership/access behavior for files associated with volumes mounted into the Pod.
Example:
spec:
securityContext:
fsGroup: 2000
Think:
Application
↓
Volume
↓
fsGroup
↓
Group-based access
Useful when an application needs group-based access to mounted volumes.
7. fsGroupChangePolicy
----------------------
Controls when Kubernetes changes volume ownership/permissions for fsGroup.
Values:
- Always
- OnRootMismatch
Always:
Always apply the ownership/permission changes.
OnRootMismatch:
Apply them only when the volume root does not already have the expected ownership/permissions.
8. supplementalGroups
---------------------
Adds additional Linux groups to the container process.
Example:
securityContext:
supplementalGroups:
- 4000
- 5000
Think:
Primary group
+
Supplementary groups
9. privileged
-------------
Runs a container in privileged mode.
Example:
securityContext:
privileged: true
Meaning:
- Gives the container highly elevated access
- Can access host devices/kernel capabilities
Generally:
privileged: false
Remember:
Privileged = highly elevated container access
10. CAPABILITIES
----------------
Linux capabilities split root privileges into individual permissions.
Example:
securityContext:
capabilities:
add:
- NET_ADMIN
drop:
- ALL
Can:
- ADD capabilities
- DROP capabilities
Example:
capabilities:
drop:
- ALL
Meaning:
Remove all capabilities.
Example:
capabilities:
add:
- NET_ADMIN
Meaning:
Give NET_ADMIN capability.
Security best practice:
Drop unnecessary capabilities and add only what is required.
11. allowPrivilegeEscalation
----------------------------
Controls whether a process can gain more privileges than its parent process.
Example:
securityContext:
allowPrivilegeEscalation: false
Meaning:
Do not allow the process to gain additional privileges.
For restricted security configurations:
allowPrivilegeEscalation: false
12. readOnlyRootFilesystem
--------------------------
Makes the container's root filesystem read-only.
Example:
securityContext:
readOnlyRootFilesystem: true
Meaning:
Container cannot write to its root filesystem.
If application needs writable temporary storage:
volumes:
- name: tmp
emptyDir: {}
Then mount it where required.
13. seccompProfile
------------------
Seccomp restricts which Linux system calls a process can make.
Example:
securityContext:
seccompProfile:
type: RuntimeDefault
Common types:
- RuntimeDefault
- Unconfined
- Localhost
RuntimeDefault:
Use container runtime's default seccomp profile.
Unconfined:
No seccomp restrictions.
Localhost:
Use a custom profile already present on the node.
14. seLinuxOptions
------------------
Configures SELinux-related security settings.
Example:
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
More advanced and depends on SELinux configuration.
15. MAJOR SECURITYCONTEXT FIELDS
--------------------------------
For CKA, remember these core fields:
Pod-level:
- runAsUser
- runAsGroup
- runAsNonRoot
- fsGroup
- fsGroupChangePolicy
- supplementalGroups
- seccompProfile
- seLinuxOptions
Container-level:
- runAsUser
- runAsGroup
- runAsNonRoot
- privileged
- allowPrivilegeEscalation
- readOnlyRootFilesystem
- capabilities
- seccompProfile
- seLinuxOptions
16. SIX CORE FIELDS FOR QUICK CKA REVISION
------------------------------------------
1. runAsUser
2. runAsGroup
3. runAsNonRoot
4. fsGroup
5. capabilities
6. allowPrivilegeEscalation
Note:
"Major 6 fields" is a useful study shortcut, NOT an official Kubernetes limit.
17. SECURE POD EXAMPLE
----------------------
apiVersion: v1
kind: Pod
metadata:
name: secure-pod
spec:
securityContext:
runAsUser: 1000
runAsGroup: 3000
runAsNonRoot: true
fsGroup: 2000
containers:
- name: app
image: nginx
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
Meaning:
Pod:
- UID = 1000
- GID = 3000
- Must not run as root
- Volume group = 2000
Container:
- Cannot escalate privileges
- Root filesystem is read-only
- All Linux capabilities are dropped
18. SECURITYCONTEXT vs POD SECURITY ADMISSION
---------------------------------------------
SecurityContext:
Defines HOW a Pod/container runs.
Examples:
- runAsUser
- privileged
- capabilities
- fsGroup
Pod Security Admission:
Defines WHAT security configuration is allowed in a namespace.
Levels:
- Privileged
- Baseline
- Restricted
Think:
Pod Security Admission
↓
"Are you ALLOWED to create this Pod?"
SecurityContext
↓
"HOW will this Pod/container run?"
19. SECURITYCONTEXT vs RBAC
---------------------------
RBAC:
Controls what a user/service account can do to Kubernetes API resources.
Examples:
- get Pods
- delete Deployments
- create Secrets
SecurityContext:
Controls how the container process runs on Linux.
Examples:
- UID
- GID
- root/non-root
- capabilities
- privileges
- filesystem
20. CKA QUICK MEMORY
--------------------
Question:
"Run container as user 1000"
→ runAsUser: 1000
"Run as non-root"
→ runAsNonRoot: true
"Set group ownership for volumes"
→ fsGroup: 2000
"Set primary group"
→ runAsGroup: 3000
"Add/remove Linux privileges"
→ capabilities
"Remove all capabilities"
→ capabilities:
drop:
- ALL
"Don't allow gaining additional privileges"
→ allowPrivilegeEscalation: false
"Make container filesystem read-only"
→ readOnlyRootFilesystem: true
"Run with elevated privileges"
→ privileged: true
"Restrict Linux system calls"
→ seccompProfile
FINAL MENTAL MODEL
------------------
Kubernetes Security
|
+----------------+----------------+
| | |
RBAC Pod Security SecurityContext
Admission |
| | |
API permissions Pod restrictions Linux process
|
+-----------------------------+----------------+
| | | | | |
runAsUser runAsGroup fsGroup capabilities privileged
| | | | |
UID GID Volume Privileges Root
|
allowPrivilegeEscalation
MOST IMPORTANT FOR CKA:
-----------------------
runAsUser
runAsGroup
runAsNonRoot
fsGroup
capabilities
allowPrivilegeEscalation
privileged
readOnlyRootFilesystem
seccompProfile
Rabbitmq Study
RABBITMQ / RHEL / KUBERNETES / DEVOPS INTERVIEW STUDY
=============================================================
PURPOSE
-------
This is the consolidated study summary of the RabbitMQ topics already learned, followed by the pending topics. Use this as the context for continuing the study later.
=============================================================
PART 1 — RABBITMQ FUNDAMENTALS
=============================================================
1. RabbitMQ is a message broker.
Main flow:
Producer -> Exchange -> Binding -> Queue -> Consumer
2. Producer normally publishes to an exchange, not directly to a queue.
3. Exchange and queue are separate objects.
Creating an exchange does NOT create a queue inside it.
A binding connects an exchange to a queue.
4. Routing key:
- A value supplied by the publisher.
- Direct exchange: exact match.
- Topic exchange: pattern matching through bindings.
- Fanout: routing key is effectively ignored.
- Headers: message headers determine routing.
- Routing key is NOT authentication.
5. Direct exchange:
Exact routing-key match.
Multiple queues can have the same routing key, so one message can be routed to multiple queues.
6. Topic exchange:
- * = exactly one word.
- # = zero or more words.
Example:
binding sms.* matches sms.otp and sms.nonotp
binding sms.# can match sms, sms.otp, sms.otp.high
7. Fanout exchange:
Sends a message to all queues bound to that exchange.
It does NOT broadcast to every queue in RabbitMQ.
8. Headers exchange:
Routes using message headers.
x-match=all -> all matching headers.
x-match=any -> any matching header.
9. Default exchange:
- Name is empty string "".
- Direct exchange.
- Every queue is automatically bound to it using the queue name as routing key.
Example:
exchange="" + routing_key="nonotp" -> queue nonotp
A custom exchange with no binding does NOT fall back to the default exchange.
10. Vhost:
Logical namespace/isolation boundary.
Contains exchanges, queues, bindings and permissions.
/ is the default vhost.
Vhost is not another RabbitMQ server.
=============================================================
PART 2 — QUEUES, DURABILITY AND DELIVERY
=============================================================
11. Durable queue:
Queue definition survives broker restart.
Durable does NOT mean replicated.
12. Persistent message:
Message is marked for durable storage.
For restart durability, normally use durable queue + persistent message.
13. Persistence/durability != HA/replication.
14. Queue types:
- Classic
- Quorum
- Stream
15. Classic queue:
Traditional queue.
Normally hosted on one node and not replicated by default.
Classic queue itself is NOT deprecated.
16. Classic queue mirroring:
Old HA mechanism.
Deprecated in RabbitMQ 3.9.
Removed in RabbitMQ 4.0.
17. Quorum queue:
Replicated queue using Raft consensus.
Usually has one leader and followers.
Each quorum queue has its own Raft group.
There is no single cluster-wide queue leader.
18. Quorum majority:
3 members -> 2/3 majority is enough to continue.
If only 1/3 remains, quorum is lost and safe progress cannot continue.
One-member quorum queue is possible but provides no HA.
19. Quorum leader failure:
Remaining majority elects a new leader.
Application does not need to track the queue leader.
20. Application connection and queue leader are different concepts.
Application can connect to Node3 while the queue leader is Node1.
21. Cluster membership does NOT automatically replicate every queue.
Classic queue data can remain on one node.
22. Classic queue on Node1:
Node2/3 know queue metadata but do not automatically have copies of the messages.
If Node1 restarts, durable/persistent data can normally recover from its disk.
Permanent loss of Node1 can mean loss/unavailability of that classic queue's data.
=============================================================
PART 3 — ACK, NACK, PREFETCH, CONNECTIONS
=============================================================
23. ACK:
Consumer tells RabbitMQ that the delivery was successfully handled.
The message is no longer outstanding for redelivery.
24. NACK:
Consumer explicitly tells RabbitMQ it could not handle the message.
requeue=true -> message can be requeued.
requeue=false -> message is not requeued; with DLX it can be dead-lettered.
25. No ACK is NOT the same as NACK.
If consumer stays connected, message remains unacknowledged.
If channel/connection closes before ACK, message can be requeued/redelivered.
26. RabbitMQ knows a message was delivered because it sent the AMQP delivery.
ACK is the consumer's confirmation of successful handling, not merely confirmation that TCP received a packet.
27. Redelivery:
Same message can be delivered again after requeue or consumer/channel failure.
Applications should be idempotent/deduplicate when duplicate business operations are possible.
RabbitMQ generally provides at-least-once style delivery, not exactly-once business processing.
28. Prefetch:
Controls how many messages can be outstanding/unacknowledged for a consumer under the applicable QoS setting.
Example prefetch=10:
RabbitMQ can have up to about 10 outstanding messages.
If 5 are ACKed, 5 slots become available and RabbitMQ can send 5 more.
Prefetch is NOT messages/second and NOT necessarily a batch size.
RabbitMQ does not wait for all 10 ACKs before replacing freed slots.
29. If prefetch=10 and consumer ACKs nothing:
10 messages remain unacknowledged.
RabbitMQ does not simply send another 10 while the same consumer remains active.
30. If consumer dies before ACK:
Unacknowledged messages can be requeued and delivered again.
31. Connection:
Actual TCP/network connection to RabbitMQ.
32. Channel:
Lightweight logical AMQP session inside a connection.
One connection can have multiple channels.
Connection failure affects its channels.
A channel can fail while the TCP connection remains available for other channels.
33. Publisher confirms:
Broker confirmation to publisher that a publish was accepted according to RabbitMQ's confirmation semantics.
Different from consumer ACK.
=============================================================
PART 4 — TTL, DLX, DLQ AND RETRY
=============================================================
34. TTL:
Time-to-live for messages.
Can be configured at queue level or per message.
No default message TTL.
Effective expiration can be affected by the shorter applicable TTL.
Physical removal may occur when RabbitMQ processes expiration, not necessarily exactly at the expiry timestamp.
35. DLX:
Dead Letter Exchange.
Not automatically enabled.
Can receive messages that are dead-lettered.
36. DLQ:
Dead Letter Queue.
Usually a normal queue bound to a DLX.
RabbitMQ does not automatically consume the DLQ.
37. Dead-letter causes include:
- NACK/reject with requeue=false
- TTL expiration
- Queue length limit
- Other configured dead-letter conditions
38. Typical flow:
Original queue -> DLX -> binding -> DLQ -> separate consumer/process
39. Retry:
Can be implemented using requeue, retry queues with TTL+DLX, or application logic.
requeue=true can cause retry storms.
Temporary failures -> retry.
Permanent/repeated failures -> DLQ.
=============================================================
PART 5 — CLUSTERING, HA AND NETWORK PARTITIONS
=============================================================
40. RabbitMQ cluster:
Multiple RabbitMQ nodes form one logical cluster.
3 nodes are common for production HA but 3 is NOT a technical requirement.
41. Cluster membership:
Nodes use Erlang/RabbitMQ distribution and the Erlang cookie.
Common cookie path:
/var/lib/rabbitmq/.erlang.cookie
42. Common cluster commands:
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@node1
rabbitmqctl start_app
rabbitmqctl cluster_status
43. rabbitmqctl vs rabbitmqadmin:
rabbitmqctl -> node/CLI administration.
rabbitmqadmin -> Management HTTP API based administration.
44. rabbitmqctl reset is destructive.
Do not casually run it in production.
45. Network partition / split brain:
Nodes are alive but cannot communicate.
In a 3-member quorum queue, a 2-node majority can continue while the isolated 1-node minority cannot safely make progress.
Quorum prevents independent minority writes.
46. Application may still have a TCP connection to an isolated minority node.
It may not immediately know that the node is a minority.
Symptoms can be failed operations, missing publisher confirms, timeouts or connection recovery.
Exact timeout/recovery depends on client/application configuration.
47. Publisher confirms help determine whether a publish was accepted.
Missing confirmation does NOT always prove message loss because the broker may have accepted the message while the confirmation was lost.
Blind retry can create duplicates.
Use idempotency/deduplication for business operations such as SMS.
=============================================================
PART 6 — FEDERATION AND SHOVEL
=============================================================
48. Federation:
Connects independent RabbitMQ brokers/clusters.
Does NOT merge them into one cluster.
Useful for multi-DC/region or independent environments.
49. Federated exchange:
Upstream exchange -> federation -> downstream exchange -> normal bindings -> queue -> consumer.
50. Federated queue:
Downstream side can obtain messages from an upstream queue.
It is NOT permanent queue replication and NOT quorum HA.
51. Shovel:
Long-running message transport mechanism.
Consumes from configured source and publishes to configured destination.
Useful for migration, integration and controlled one-way message movement.
Source/destination can be queue/exchange depending configuration.
52. Federation vs Shovel:
Federation -> broker/resource-level ongoing relationship between independent environments.
Shovel -> explicit source-to-destination message transport.
Neither is HA/replication.
=============================================================
PART 7 — SECURITY
=============================================================
53. Authentication:
Answers: Who are you?
RabbitMQ can use local users/passwords or LDAP.
54. Authorization:
Answers: What can you access/do?
Controlled through vhosts and permissions.
55. RabbitMQ permissions:
- configure
- write
- read
56. Useful commands:
rabbitmqctl list_users
rabbitmqctl list_vhosts
rabbitmqctl list_permissions -p /
rabbitmqctl list_user_permissions <user>
57. Example:
rabbitmqctl set_permissions -p /sms smsapp ".*" ".*" ".*"
This grants broad permissions; production should use least privilege.
58. TLS:
5672 -> normal AMQP
5671 -> AMQP over TLS
15672 -> Management HTTP
15671 -> Management HTTPS
59. TLS provides:
Confidentiality
Integrity
Server authentication
60. CA:
Certificate Authority signs certificates.
Java truststore normally contains trusted CA certificates.
61. Truststore:
"Who do I trust?"
Used to verify RabbitMQ server certificate.
62. Keystore:
"Who am I?"
Contains client certificate/private key when client identity is needed, such as mTLS.
63. Normal one-way TLS:
RabbitMQ presents server certificate.
Java client verifies it using truststore.
Client certificate is not normally required.
64. mTLS:
Both sides authenticate using certificates.
RabbitMQ can require client certificates.
Java client uses keystore for client identity and truststore for trusted CA.
mTLS does not replace RabbitMQ authorization.
65. Example RabbitMQ TLS settings:
listeners.ssl.default = 5671
ssl_options.cacertfile = /etc/rabbitmq/certs/ca.pem
ssl_options.certfile = /etc/rabbitmq/certs/server.pem
ssl_options.keyfile = /etc/rabbitmq/certs/server.key
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
66. TLS troubleshooting:
Network -> 5671 -> certificate chain/truststore -> hostname/SAN -> TLS versions/ciphers -> mTLS client cert -> RabbitMQ authentication -> vhost/permissions.
67. LDAP:
RabbitMQ 4.x uses rabbitmq_auth_backend_ldap plugin.
Enable:
rabbitmq-plugins enable rabbitmq_auth_backend_ldap
68. LDAP can authenticate while RabbitMQ internal backend handles authorization:
auth_backends.1.authn = ldap
auth_backends.1.authz = internal
69. LDAP authorization can use:
vhost_access_query
resource_access_query
topic_access_query
tag_queries
Exact advanced.config syntax should be verified against the target RabbitMQ version before production.
=============================================================
PART 8 — BACKUP, RESTORE AND MIGRATION
=============================================================
70. Definitions backup:
rabbitmqctl export_definitions /backup/rabbitmq-definitions.json
71. Restore:
rabbitmqctl import_definitions /backup/rabbitmq-definitions.json
72. Definitions include:
vhosts, users, permissions, exchanges, queues, bindings, policies, parameters, etc.
73. Definitions backup does NOT contain queued message bodies.
74. There is no general native "dump one queue's messages and restore it" command.
Message-level export can be done with application logic, Management API get/requeue workflows, shovel, or consume/re-publish strategies depending requirements.
75. Do not casually copy a live /var/lib/rabbitmq directory as a consistent backup.
For message-store/data backup, follow the RabbitMQ-supported procedure and normally stop the node for a consistent filesystem-level backup.
76. Blue-Green/fresh-cluster migration:
- Move application to DR
- Stop/quiesce producers
- Verify production queues are drained
- Export definitions
- Backup configuration/certs
- Stop old cluster
- Provision clean RabbitMQ 4.x + compatible Erlang
- Build new cluster
- Import definitions
- Validate
- Switch application back
77. This is a fresh-cluster/Blue-Green migration, NOT a rolling upgrade.
78. Rolling upgrade:
Only use a source->target version path explicitly supported by RabbitMQ.
Mixed versions may temporarily coexist when supported.
Upgrade one node, verify, continue.
Feature flags should be enabled only according to the supported upgrade process.
=============================================================
PART 9 — START/STOP, HEALTH CHECKS AND DATA DIRECTORY
=============================================================
79. Useful commands:
systemctl status rabbitmq-server
systemctl stop rabbitmq-server
systemctl restart rabbitmq-server
rabbitmqctl stop
rabbitmq-diagnostics status
rabbitmq-diagnostics ping
rabbitmq-diagnostics alarms
rabbitmq-diagnostics listeners
rabbitmqctl cluster_status
80. Queue/consumer checks:
rabbitmqctl list_queues name type messages messages_ready messages_unacknowledged
rabbitmqctl list_connections
rabbitmqctl list_channels
rabbitmqctl list_consumers
81. RabbitMQ data directory commonly:
/var/lib/rabbitmq
82. Custom data directory can be configured using:
RABBITMQ_MNESIA_BASE=/app/rabbitmq
usually in /etc/rabbitmq/rabbitmq-env.conf
83. User's lab:
NODENAME=rabbit@node8
RABBITMQ_MNESIA_BASE=/app/rabbitmq
84. Active Mnesia directory can be checked:
rabbitmqctl eval 'rabbit_mnesia:dir().'
85. Important:
Stop RabbitMQ before moving/copying node data.
Preserve ownership/permissions.
Do not blindly merge Mnesia databases.
Do not delete the old directory until the new setup is validated.
Custom paths on RHEL/Rocky may require SELinux context handling.
Never share one RabbitMQ data directory between multiple RabbitMQ nodes.
=============================================================
PART 10 — MEMORY, DISK, CPU AND PERFORMANCE
=============================================================
86. RabbitMQ uses both memory and disk.
Do not simplify this to "messages are always in RAM and then move to disk."
Internal storage behavior is more sophisticated.
87. Memory alarm:
RabbitMQ has a memory high-watermark/protection mechanism.
When memory pressure reaches the threshold, RabbitMQ can raise a memory alarm and apply publisher flow control.
88. Commands:
rabbitmq-diagnostics alarms
rabbitmq-diagnostics memory_breakdown
free -h
top
89. free -h is OS-wide, not RabbitMQ-specific.
It includes filesystem cache, kernel memory and other processes.
90. Disk alarm:
RabbitMQ protects itself when available disk space becomes too low.
Publishers can be blocked/slowed.
91. Commands:
df -h
df -h /app/rabbitmq
rabbitmq-diagnostics alarms
92. Disk capacity vs disk performance:
df -h -> how much space is available
iostat -xz 1 -> disk I/O/performance
93. CPU:
top
mpstat -P ALL 1
94. Network:
sar -n DEV 1
ip -s link
tcpdump
95. Performance depends on:
CPU
memory
disk I/O
network
message size
message rate
persistence
publisher confirms
queue type
quorum replication
consumers
prefetch
downstream dependencies
96. Producer rate vs consumer rate:
If producer > consumer, backlog grows.
backlog growth approximately = producer rate - consumer rate.
97. Example:
Producer 10,000 msg/sec
Consumer 7,000 msg/sec
Backlog grows about 3,000 msg/sec.
98. Message size matters:
200 KB × 5,000 msg/sec ≈ 1 GB/sec raw payload.
This is before protocol/replication/storage overhead.
99. Connection capacity:
No universal connections/sec limit.
Concurrent connections and connection-establishment rate are different.
Reuse long-lived connections.
OS file descriptors, CPU, RAM, TLS and workload affect capacity.
100. User's example hardware:
8 GB RAM, 500 GB disk, 8 CPU cores, 10 Gbps NIC.
Previous rough planning estimate: 5k–10k concurrent connections per node as an initial planning range, not a RabbitMQ limit; benchmark actual workload.
101. 80 million messages:
Storage depends on average message size.
80M × 200 KB ≈ 16 TB raw payload.
Message count alone is insufficient for capacity planning.
=============================================================
PART 11 — QUEUE TROUBLESHOOTING AND PERFORMANCE
=============================================================
102. messages_ready:
Messages waiting to be delivered.
103. messages_unacknowledged:
Messages delivered to consumers but not yet ACKed.
104. High ready + low unacked:
Consumers absent, slow, disconnected or unable to keep up.
105. Low ready + high unacked:
Consumer has messages but is slow/stuck/not ACKing.
106. Both zero:
Queue is empty; could be healthy or producer may not be publishing.
107. Consumer throughput:
Depends on number of consumers and processing time, but downstream DB/API/workload often becomes the real bottleneck.
108. Example:
100 ms/message ≈ 10 msg/sec/consumer.
10 consumers ≈ 100 msg/sec in a simple idealized case.
109. High prefetch:
Can produce many unacknowledged messages, memory pressure and uneven work distribution.
Do not blindly increase it.
110. Low prefetch:
Can reduce pipeline utilization if consumer processing has latency.
111. Good tuning method:
Measure consumer processing time, ready/unacked counts and throughput.
Adjust prefetch gradually while monitoring throughput, latency, memory and fairness.
112. Message-store optimization:
Find why messages accumulate.
Avoid unnecessary huge payloads.
Consider external object/file storage plus a reference in RabbitMQ for very large payloads.
Optimize consumers instead of simply increasing storage.
113. Do not purge production queues without confirming business impact and approval.
=============================================================
PART 12 — PUBLISHER THROUGHPUT
=============================================================
114. Publisher throughput can be limited by:
CPU
network
disk I/O
message size
persistence
publisher confirms
quorum replication
connections/channels
memory/disk alarms
downstream/storage pressure
115. Use long-lived connections.
Do not create a connection per message.
116. Publisher confirms improve delivery assurance but add overhead.
117. Quorum queues add replication/consensus/storage/network work compared with a non-replicated classic queue.
118. If publishing becomes slow:
Check CPU, disk I/O, network, alarms, message rate/size, confirms, queue type and client behavior.
=============================================================
PART 13 — CONNECTION AND CHANNEL TROUBLESHOOTING
=============================================================
119. Connection troubleshooting sequence:
DNS/hostname
-> network
-> firewall
-> port
-> RabbitMQ listener
-> TLS
-> authentication
-> vhost
-> authorization
-> resource
120. Commands:
getent hosts rabbit1
nslookup rabbit1
nc -vz rabbit1 5672
nc -vz rabbit1 5671
rabbitmq-diagnostics listeners
ss -lntp
firewall-cmd --list-all
rabbitmqctl list_connections
121. Heartbeats:
AMQP connection-liveness mechanism.
Helps detect failed/inactive connections.
Not an application health check.
122. Cluster networking is separate from application connectivity.
Application -> RabbitMQ on 5672/5671 is one path.
RabbitMQ node-to-node communication is another.
123. Channel troubleshooting:
Check list_channels and logs.
Look for PRECONDITION_FAILED, ACCESS_REFUSED, NOT_FOUND.
Common causes include declaring a queue/exchange with conflicting properties, invalid operations and permission issues.
124. Important distinction:
Connection failure affects channels belonging to that connection.
Channel failure does not necessarily bring down the whole TCP connection.
=============================================================
PART 14 — THREAD DUMPS
=============================================================
125. Thread dump:
Snapshot of Java application threads at a moment in time.
126. Important because:
RabbitMQ itself is Erlang/BEAM.
Java thread dumps are primarily for Tomcat/Java publisher/consumer applications.
127. Useful commands:
jps -l
ps -ef | grep java
jstack <PID> > thread_dump.txt
kill -3 <PID>
128. Taking multiple dumps:
Example:
dump 1 at t=0
dump 2 at t=5 sec
dump 3 at t=10 sec
If the same threads remain stuck in the same place, it is stronger evidence of a persistent problem.
129. Thread states:
RUNNABLE -> running or ready to run; does not always mean actively executing on CPU.
BLOCKED -> waiting to acquire a Java monitor lock held by another thread.
WAITING -> waiting indefinitely for an event/notification.
TIMED_WAITING -> waiting with a timeout.
130. Lock contention:
One thread holds a lock and another waits.
May be temporary.
131. Deadlock:
Circular lock dependency.
Example:
Thread A holds Lock1 and wants Lock2.
Thread B holds Lock2 and wants Lock1.
JVM thread dump may report a Java-level deadlock.
132. RabbitMQ connection to Java troubleshooting:
RabbitMQ can be healthy while Java consumer threads are blocked on DB, network, locks or other work.
This can delay ACKs and increase unacknowledged messages.
=============================================================
PART 15 — HEAP DUMPS
=============================================================
133. Thread dump vs heap dump:
Thread dump -> what threads are doing.
Heap dump -> what Java objects are consuming/retaining memory.
134. Java heap:
Memory area where Java objects are allocated.
135. Java memory leak:
Application unintentionally retains references to objects it no longer needs, preventing GC from reclaiming them.
136. Useful commands:
jps -l
jmap -dump:live,format=b,file=/tmp/heap.hprof <PID>
jcmd <PID> GC.heap_dump /tmp/heap.hprof
137. Heap dump analysis asks:
Which classes/objects consume the most memory?
What is retaining them?
Why can't GC reclaim them?
138. Tools:
Eclipse MAT and other heap-analysis tools.
139. RabbitMQ relation:
Java consumer memory leak -> GC pressure -> application slows -> ACKs delayed -> RabbitMQ unacked count increases.
RabbitMQ may not be the root cause.
=============================================================
PART 16 — TROUBLESHOOTING MINDSET
=============================================================
140. Never immediately restart/purge when a problem appears.
141. Use layered troubleshooting:
Application
-> connection
-> channel
-> RabbitMQ node
-> cluster
-> queue
-> consumer/publisher
-> CPU/memory/disk/network
-> downstream dependencies
142. Always correlate:
- queue ready/unacked
- producer rate
- consumer rate
- CPU
- memory
- disk capacity
- disk I/O
- network
- application logs
- RabbitMQ logs
143. A RabbitMQ symptom does not necessarily mean RabbitMQ is the root cause.
Example:
Java DB lock -> consumer stuck -> no ACK -> RabbitMQ unacked grows.
=============================================================
PART 17 — PENDING TOPICS
=============================================================
A. Java/JVM/RabbitMQ troubleshooting
------------------------------------
1. GC troubleshooting
2. OutOfMemoryError
3. GC logs and GC pauses
4. Heap dump practical analysis
5. RabbitMQ support bundle
6. Log-based RCA
7. RCA methodology
8. Complete production incident workflow
9. Deep queue troubleshooting scenarios
10. Deep consumer troubleshooting scenarios
11. Deep publisher troubleshooting scenarios
B. RabbitMQ operations/scripts
------------------------------
12. Health-check shell script
13. Monitoring script
14. Start/stop/restart script
15. Maintenance scripts
16. Automated queue/consumer checks
C. RHEL/Linux
-------------
17. RHEL networking fundamentals
18. Ports and firewall
19. DNS/hostname troubleshooting
20. Routing/network troubleshooting
21. Storage troubleshooting
22. LVM
23. Filesystem troubleshooting
24. Disk I/O troubleshooting
D. Other JD topics
------------------
25. Caching
26. IMDG
27. NoSQL
28. Kubernetes architecture
29. Pod lifecycle
30. Deployments
31. Kubernetes + RabbitMQ
32. CI/CD
33. DevOps concepts
E. Final interview preparation
------------------------------
34. RabbitMQ scenario-based troubleshooting
35. Failure/HA scenarios
36. Security/TLS/LDAP scenarios
37. Backup/migration/upgrade scenarios
38. RHEL troubleshooting scenarios
39. Kubernetes/RabbitMQ scenarios
40. Full mock interview
41. RCA-based interview questions
=============================================================
CONTINUE FROM HERE
=============================================================
Next topic:
GC troubleshooting + OutOfMemoryError
When continuing, teach each topic:
1. Simple concept
2. Why it happens
3. Practical commands
4. Production example
5. Interview-ready answer
6. Common mistakes/traps
7. Then ask/continue to the next topic only after the user says OK.
=============================================================
COPY-PASTE CONTINUATION PROMPT
=============================================================
I am preparing for a RabbitMQ/RHEL/Kubernetes/DevOps interview based on a JD involving RabbitMQ administration, clustering, HA, federation, shovels, TLS/LDAP, troubleshooting, RCA, shell scripting, RHEL networking/storage/LVM, caching, IMDG, NoSQL, Kubernetes, CI/CD and DevOps.
Use the following study progress as context:
COMPLETED RABBITMQ:
- RabbitMQ architecture and Producer -> Exchange -> Binding -> Queue -> Consumer
- Direct, Topic, Fanout, Headers exchanges
- Default exchange
- Routing keys and bindings
- Vhosts
- Queues
- Durable queues
- Persistent messages
- ACK/NACK/manual/auto ACK
- Redelivery and at-least-once behavior
- Prefetch/QoS
- Connections vs channels
- Publisher confirms
- TTL
- DLX/DLQ
- Retry
- RabbitMQ clustering
- Cluster vs replication
- Quorum queues
- Raft, leader/followers, majority and elections
- Classic vs quorum
- Classic queue mirroring deprecation/removal
- Federation
- Federated exchange/queue
- Shovel
- Federation vs Shovel
- HA
- Node failures
- Quorum failures
- Network partitions/split-brain
- Application connection recovery concepts
- Authentication
- Authorization
- Users/vhosts/permissions
- configure/write/read permissions
- TLS/SSL
- Certificates
- Truststore vs keystore
- mTLS
- LDAP authentication and internal authorization
- Backup/restore
- Definitions export/import
- Message-store/data backup concepts
- Blue-Green migration
- Rolling upgrade concepts and compatibility
- Start/stop/restart
- Health checks
- RabbitMQ data directory/Mnesia migration
- Memory alarms
- Disk alarms
- CPU/resource troubleshooting
- Network troubleshooting
- Performance troubleshooting
- Queue backlog troubleshooting
- Consumer throughput
- Publisher throughput
- Message-store optimization
- Queue performance tuning
- Prefetch real-world behavior
- Connection troubleshooting
- Channel troubleshooting
- Thread dumps
- Java thread states
- Lock contention/deadlock
- Heap dumps and Java memory leaks
IMPORTANT PREFETCH UNDERSTANDING:
If prefetch=10 and one consumer receives 10 messages, RabbitMQ allows up to 10 outstanding/unacknowledged messages for that consumer. It does NOT wait for all 10 ACKs before sending more. If 5 ACKs arrive, 5 slots become available and RabbitMQ can send 5 more. If no ACKs arrive and the consumer remains alive, RabbitMQ does not simply send another 10. If the consumer/channel dies before ACK, the messages can be requeued/redelivered.
IMPORTANT ACK UNDERSTANDING:
RabbitMQ knows it delivered a message because it sent the AMQP delivery. ACK is the consumer's confirmation that processing/handling succeeded. No ACK does not equal NACK. If the consumer dies before ACK, the message can be redelivered. Consumers should handle duplicates/idempotency.
IMPORTANT TROUBLESHOOTING MODEL:
Application -> connection -> channel -> RabbitMQ node -> cluster -> queue -> consumer/publisher -> CPU/memory/disk/network -> downstream dependencies.
PENDING TOPICS:
1. GC troubleshooting
2. OutOfMemoryError
3. GC logs and GC pauses
4. Practical heap dump analysis
5. RabbitMQ support bundle
6. Log-based RCA
7. RCA methodology
8. Complete production incident workflow
9. Deep queue troubleshooting scenarios
10. Deep consumer troubleshooting scenarios
11. Deep publisher troubleshooting scenarios
12. Health-check shell script
13. Monitoring script
14. Start/stop/restart script
15. Maintenance scripts
16. Automated queue/consumer checks
17. RHEL networking
18. Ports/firewall
19. DNS/hostname
20. Routing/network troubleshooting
21. Storage troubleshooting
22. LVM
23. Filesystem troubleshooting
24. Disk I/O
25. Caching
26. IMDG
27. NoSQL
28. Kubernetes architecture
29. Pod lifecycle
30. Deployments
31. Kubernetes + RabbitMQ
32. CI/CD
33. DevOps
34. RabbitMQ scenario-based troubleshooting
35. HA/failure scenarios
36. Security/TLS/LDAP scenarios
37. Backup/migration/upgrade scenarios
38. RHEL troubleshooting scenarios
39. Kubernetes/RabbitMQ scenarios
40. Full mock interview
41. RCA-based interview questions
Continue from:
GC troubleshooting + OutOfMemoryError
Teaching style:
- Explain simply first.
- Use practical examples.
- Use RabbitMQ/Tomcat examples where useful.
- Explain WHY, not just commands.
- Give important Linux/RabbitMQ commands.
- Clearly distinguish RabbitMQ problems from Java application problems.
- Give an interview-ready answer after each major topic.
- Point out common interview traps/corrections.
- Do not skip fundamentals.
- Proceed topic by topic and wait for my "OK" before moving to the next topic.
Rabbitmq Fast Revision
GOAL: Understand RabbitMQ internals, not memorize commands.
Producer ↓ Exchange ↓ routing key + binding Queue ↓ Consumer
Producer normally knows:
Producer does NOT need to know:
RabbitMQ admin defines queues + bindings.
Exchange = routing decision Queue = stores messages Consumer = processes messages
DIRECT: Exact routing-key match.
topic.sms → queue A topic.email → queue B
IMPORTANT: Direct exchange does NOT support * or # wildcards. "sms.*" is literal.
TOPIC: Pattern matching.
sms.* → sms.email sms.* → sms.email.audit ❌
sms.# → sms sms.# → sms.email sms.# → sms.email.audit
FANOUT: Ignores routing key. Sends to every bound queue.
HEADERS: Routes using message headers. x-match = all / any
DEFAULT EXCHANGE: Routing key is queue name. Producer can directly publish to a queue.
CLASSIC: Simple/lightweight queue.
QUORUM: Replicated HA queue. Uses Raft.
Example 3-member quorum:
Leader + Follower + Follower
3 members → tolerate 1 failure 5 members → tolerate 2 failures
IMPORTANT: RabbitMQ Cluster ≠ automatic queue replication.
Cluster shares topology/metadata. Queue replication depends on queue type/configuration.
STREAM: Designed for high-throughput stream-like workloads.
Cluster = multiple RabbitMQ nodes acting as one logical cluster.
Example:
rabbit1 rabbit2 rabbit3
Cluster shares metadata such as:
NOT every queue/message is automatically replicated.
Quorum queue provides queue-level replication.
Node failure: Quorum leader fails → remaining members elect new leader → clients connected to failed node lose connection → application must reconnect/recover
Cluster failover ≠ client connection failover.
5672 = AMQP 5671 = AMQP TLS 15672 = Management HTTP/API 15671 = Management HTTPS 4369 = EPMD 25672 = Erlang distribution (commonly)
Distribution port range can be configured, e.g.: 35672-35682
Always verify actual configuration before opening firewall ports.
Useful:
ss -lntp rabbitmq-diagnostics listeners epmd -names
Connection = TCP connection.
Channel = lightweight logical session inside connection.
One connection: ├── Channel 1 ├── Channel 2 └── Channel 3
Avoid creating a new TCP connection for every message/request.
Connection failure: → all channels on it are affected.
Channel failure: → other channels can continue.
Publisher Confirm: RabbitMQ confirms broker accepted the publish.
It does NOT mean: "Consumer successfully processed message."
Publisher Confirm ≠ Consumer ACK
Flow:
Producer ↓ publish RabbitMQ ↓ confirm Producer
Consumer: RabbitMQ ↓ message Consumer ↓ ACK RabbitMQ
ACK: Processing successful.
NACK/reject: Processing failed.
requeue=true: Message can be delivered again.
requeue=false: No same-queue requeue. Can go to DLX if configured.
BAD: NACK + requeue=true forever → redelivery loop.
Consumer count = number of workers.
Prefetch = outstanding message window per consumer.
Example:
5 consumers prefetch = 10
≈ 5 × 10 = 50 outstanding messages
Higher prefetch:
prefetch=1: Usually more balanced/fair.
Same queue + multiple consumers:
Queue
/ | \
C1 C2 C3
Messages are distributed among consumers.
Multiple queues with matching bindings:
Exchange ├── Queue A → Consumer └── Queue B → Consumer
Message can reach BOTH queues.
That is different from competing consumers.
Single consumer: Usually queue order.
Multiple consumers: No global processing-order guarantee.
Requeue/failure/prefetch can affect application-observed order.
Common properties:
delivery_mode content_type message_id correlation_id reply_to expiration headers
Routing key is NOT a message property. It is used by the exchange for routing.
DLX = Dead Letter EXCHANGE DLQ = Dead Letter QUEUE
DLX is NOT itself a queue.
Example:
Main Queue ↓ failure DLX ↓ DLQ
Typical reasons:
Main Exchange ↓ Main Queue ↓ failure Retry Exchange ↓ Retry Queue TTL = 30 sec ↓ expiry Main Exchange ↓ Main Queue
IMPORTANT: DLX ≠ retry automatically.
Need explicit retry design.
Avoid infinite retry. Use maximum attempts/business logic.
Durable queue: Queue survives broker restart.
Persistent message: Message is stored persistently according to RabbitMQ's persistence mechanisms.
For durable messaging: Use durable queue + persistent messages appropriately.
Quorum queues additionally provide replication.
Persistence ≠ replication.
AMQP transactions:
tx.select publish tx.commit or tx.rollback
Publisher confirms are generally preferred for performance-sensitive publishing.
RabbitMQ RPC commonly uses:
reply_to correlation_id
Request: Producer → request queue
Response: Consumer → reply queue
correlation_id identifies which response belongs to which request.
Authentication: WHO are you?
Authorization: WHAT are you allowed to do?
Local user:
rabbitmqctl add_user appuser password
Vhost:
rabbitmqctl add_vhost /production
Permissions:
rabbitmqctl set_permissions -p /production appuser "." "." ".*"
Permission order:
configure write read
User tags mainly affect management/UI capabilities.
LDAP can provide authentication.
RabbitMQ still needs authorization rules/permissions.
Can configure:
internal LDAP
LDAP: 389
LDAPS: 636
Keep a local break-glass admin when appropriate.
AMQP TLS: 5671
Important files:
CA certificate Server certificate Private key
Typical:
ssl_options.cacertfile ssl_options.certfile ssl_options.keyfile ssl_options.verify
mTLS:
verify_peer fail_if_no_peer_cert = true
"unable to get local issuer certificate" → client cannot build a trusted certificate chain.
curl -k: Disables certificate verification. Useful only for testing, NOT a security fix.
Hostname/SAN must also match.
Federation connects INDEPENDENT RabbitMQ brokers/clusters.
Cluster: One logical RabbitMQ cluster.
Federation: Independent brokers connected together.
Federation mental model:
Upstream ↓ Federation ↓ Downstream
Federated queue:
Upstream Queue ↓ Federation ↓ Downstream Queue ↓ Application Consumer
Federated exchange:
Upstream Exchange ↓ Federation ↓ Downstream Exchange ↓ Queue ↓ Consumer
Downstream federated queue behaves somewhat like a consumer of upstream.
SHOVEL:
Source Queue ↓ Shovel ↓ Destination Exchange/Queue
Easy memory:
FEDERATION = topology/following relationship SHOVEL = explicit message movement
Upstream definition: WHERE/HOW to connect.
Federation policy: WHAT object to federate.
Policy is configured on DOWNSTREAM.
If upstream is unavailable:
Federation link breaks ↓ Reconnect attempts ↓ Already transferred downstream messages remain downstream ↓ Messages existing only upstream remain upstream
Downstream can continue processing messages it already has.
rabbitmq-diagnostics ping → Is RabbitMQ responding?
rabbitmq-diagnostics status → What is happening on this node?
rabbitmqctl cluster_status → What does the cluster look like?
rabbitmq-diagnostics listeners → Which ports is RabbitMQ listening on?
rabbitmqctl list_queues name messages messages_ready messages_unacknowledged
READY: Waiting for delivery.
UNACKED: Delivered to consumer but ACK not received.
MOST IMPORTANT:
READY ↑ → consumer capacity/availability problem likely.
UNACKED ↑ → consumers received messages but aren't ACKing fast enough.
Publish rate: Producer → RabbitMQ
Delivery rate: RabbitMQ → Consumer
ACK rate: Consumer → RabbitMQ
Redelivery rate: Messages being delivered again.
Example:
Publish = 1000/s Consumer/ACK = 400/s
→ backlog grows.
Example:
Delivery = 500/s ACK = 100/s
→ UNACKED grows.
Always correlate: READY + UNACKED + rates.
rabbitmqctl list_connections
Connection = TCP.
Too many connections may indicate:
Do not create a new RabbitMQ TCP connection for every message.
rabbitmqctl list_channels
Channel = lightweight logical session inside a connection.
Useful when investigating:
rabbitmqctl list_consumers
Check:
Scenario:
READY = 50,000 UNACKED = 0 Consumers = 0
→ Consumer unavailable/configuration/connectivity problem likely.
Scenario:
READY = increasing Consumers exist
→ Consumer capacity may be lower than producer rate.
Scenario:
UNACKED = increasing
→ Consumer processing/ACK problem likely.
RabbitMQ protects itself from resource exhaustion.
Memory alarm: Memory usage exceeds configured threshold.
Disk alarm: Available disk space becomes too low.
Check:
rabbitmq-diagnostics alarms
Important:
RabbitMQ process UP ≠ RabbitMQ accepting publishes normally
A resource alarm can cause publishers to be blocked.
Message flow:
Queue ↓ Consumer receives ↓ UNACKED ↓ Application processes ↓ ACK ↓ RabbitMQ removes message
If consumer is slow: UNACKED ↑
If DB/API is slow: UNACKED ↑
If consumer hangs: UNACKED ↑
If consumer crashes before ACK: Message can be redelivered.
High prefetch can increase outstanding messages.
Problem: "RabbitMQ messages are stuck."
STEP 1: Is RabbitMQ alive?
rabbitmq-diagnostics ping
STEP 2: Node healthy?
rabbitmq-diagnostics status
STEP 3: Cluster healthy?
rabbitmqctl cluster_status
STEP 4: Check listeners/network.
rabbitmq-diagnostics listeners ss -lntp
STEP 5: Check queue.
messages_ready messages_unacknowledged
STEP 6: Check rates.
publish delivery ACK redelivery
STEP 7: Check consumers.
rabbitmqctl list_consumers
STEP 8: Check connections/channels.
rabbitmqctl list_connections rabbitmqctl list_channels
STEP 9: Check resource alarms.
rabbitmq-diagnostics alarms
STEP 10: Check consumer application logs, DB/API dependencies and processing time.
READY ↑ + UNACKED low → Consumers unavailable/slow capacity.
READY ↑ + UNACKED ↑ → Incoming rate > processing capacity.
READY low + UNACKED ↑ → Consumer processing/ACK problem.
Consumers = 0 → Consumer/application/config/network problem.
RabbitMQ UP + publishing blocked → Check resource alarms.
Consumer crashes before ACK → Unacked messages can be redelivered.
Redelivery ↑↑ → Check failures/requeue loops.
When troubleshooting RabbitMQ:
PRODUCER ↓ PUBLISH RATE ↓ EXCHANGE ↓ ROUTING/BINDING ↓ QUEUE ↓ READY ↓ DELIVERY ↓ CONSUMER ↓ PROCESSING ↓ ACK ↓ MESSAGE COMPLETE
At every stage ask:
"Where is the message currently stuck?"
rabbitmq-diagnostics ping rabbitmq-diagnostics status rabbitmq-diagnostics listeners rabbitmq-diagnostics alarms
rabbitmqctl cluster_status
rabbitmqctl list_queues rabbitmqctl list_consumers rabbitmqctl list_connections rabbitmqctl list_channels
ss -lntp epmd -names journalctl -u rabbitmq-server
RabbitMQ Cluster: Multiple nodes forming one logical cluster.
Quorum Queue: Replicated queue using Raft.
Exchange: Routes messages to queues.
Binding: Defines exchange → queue routing relationship.
Routing key: Input used by exchange for routing.
READY: Waiting for consumer delivery.
UNACKED: Delivered but not ACKed.
Prefetch: Limits outstanding messages per consumer.
Publisher Confirm: Broker accepted publish.
Consumer ACK: Consumer successfully processed/accepted responsibility.
DLX: Exchange receiving dead-lettered messages.
DLQ: Queue storing dead-lettered messages.
Federation: Connects independent RabbitMQ brokers/clusters.
Shovel: Moves messages explicitly from source to destination.
Authentication: Who are you?
Authorization: What can you do?
TLS: Protects communication.
DON'T RESTART FIRST.
CHECK:
HEALTH ↓ CLUSTER ↓ QUEUE ↓ READY / UNACKED ↓ RATES ↓ CONSUMERS ↓ CONNECTIONS ↓ ALARMS ↓ APPLICATION / DB / NETWORK
Rabbitmq Command Cheat Sheet
==================================================
Start: systemctl start rabbitmq-server
Stop: systemctl stop rabbitmq-server
Restart: systemctl restart rabbitmq-server
Status: systemctl status rabbitmq-server
Enable at boot: systemctl enable rabbitmq-server
Enable + start: systemctl enable --now rabbitmq-server
Check RabbitMQ alive: rabbitmq-diagnostics ping
Node status: rabbitmq-diagnostics status
Check listeners: rabbitmq-diagnostics listeners
Check alarms: rabbitmq-diagnostics alarms
Cluster status: rabbitmqctl cluster_status
Stop RabbitMQ application: rabbitmqctl stop_app
Start RabbitMQ application: rabbitmqctl start_app
Join cluster: rabbitmqctl join_cluster rabbit@rabbit1
Leave/reset node: rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app
IMPORTANT: reset removes the node's existing cluster membership/data. Use carefully.
Add user: rabbitmqctl add_user appuser 'StrongPassword'
List users: rabbitmqctl list_users
Change password: rabbitmqctl change_password appuser 'NewPassword'
Delete user: rabbitmqctl delete_user appuser
Administrator: rabbitmqctl set_user_tags appuser administrator
Monitoring: rabbitmqctl set_user_tags appuser monitoring
Remove tags: rabbitmqctl set_user_tags appuser
Create: rabbitmqctl add_vhost /production
List: rabbitmqctl list_vhosts
Delete: rabbitmqctl delete_vhost /production
Set permissions:
rabbitmqctl set_permissions -p /production appuser "." "." ".*"
Order:
configure write read
List permissions:
rabbitmqctl list_permissions -p /production
List user's permissions:
rabbitmqctl list_user_permissions appuser
Clear permissions:
rabbitmqctl clear_permissions -p /production appuser
List exchanges:
rabbitmqctl list_exchanges
Create using rabbitmqadmin:
rabbitmqadmin declare exchange
name=smsapp
type=topic
durable=true
Direct:
rabbitmqadmin declare exchange
name=my.direct
type=direct
durable=true
Fanout:
rabbitmqadmin declare exchange
name=my.fanout
type=fanout
durable=true
Headers:
rabbitmqadmin declare exchange
name=my.headers
type=headers
durable=true
List queues:
rabbitmqctl list_queues
Detailed queue information:
rabbitmqctl list_queues
name messages messages_ready messages_unacknowledged
Create queue:
rabbitmqadmin declare queue
name=sms.email
durable=true
Delete queue:
rabbitmqadmin delete queue name=sms.email
List bindings:
rabbitmqctl list_bindings
Create direct binding:
rabbitmqadmin declare binding
source=my.direct
destination=sms.email
destination_type=queue
routing_key=sms.email
Create topic binding:
rabbitmqadmin declare binding
source=smsapp
destination=sms.email
destination_type=queue
routing_key=sms.email
Topic wildcard:
routing_key='sms.*'
or:
routing_key='sms.#'
IMPORTANT:
Direct exchange: Exact match.
Topic exchange:
Publish:
rabbitmqadmin publish
exchange=smsapp
routing_key=sms.email
payload="Hello RabbitMQ"
Default exchange:
rabbitmqadmin publish
exchange=
routing_key=sms.email
payload="Hello"
List consumers:
rabbitmqctl list_consumers
Useful for checking:
List connections:
rabbitmqctl list_connections
Useful information:
List channels:
rabbitmqctl list_channels
Useful for:
Basic:
rabbitmqctl list_queues
Important:
rabbitmqctl list_queues
name messages messages_ready messages_unacknowledged
Meaning:
messages = total messages
messages_ready = waiting for consumers
messages_unacknowledged = delivered but not ACKed
List policies:
rabbitmqctl list_policies
Set policy example:
rabbitmqctl set_policy my-policy
"^sms."
'{"message-ttl":60000}'
--apply-to queues
Delete:
rabbitmqctl clear_policy my-policy
List parameters:
rabbitmqctl list_parameters
Used for things such as:
Check plugin:
rabbitmq-plugins list | grep federation
Enable:
rabbitmq-plugins enable rabbitmq_federation
Enable management federation UI/plugin if needed:
rabbitmq-plugins enable rabbitmq_federation_management
List parameters:
rabbitmqctl list_parameters
Federation status:
rabbitmqctl federation_status
Set upstream example:
rabbitmqctl set_parameter federation-upstream clusterA
'{"uri":"amqp://user:password@192.168.241.131:5672/%2f","prefetch-count":1000,"reconnect-delay":5}'
Federation policy example:
rabbitmqctl set_policy federate-sms
"^smsapp$"
'{"federation-upstream-set":"all"}'
--priority 10
--apply-to exchanges
List plugins:
rabbitmq-plugins list
Enable plugin:
rabbitmq-plugins enable PLUGIN_NAME
Disable:
rabbitmq-plugins disable PLUGIN_NAME
Check listening ports:
ss -lntp
Filter RabbitMQ ports:
ss -lntp | grep -E
'4369|5671|5672|15671|15672|25672'
Check RabbitMQ listeners:
rabbitmq-diagnostics listeners
Test AMQP port:
nc -zv 192.168.241.131 5672
Test AMQP TLS:
nc -zv 192.168.241.131 5671
Test Erlang distribution:
nc -zv 192.168.241.131 25672
Check EPMD:
epmd -names
AMQP TLS:
openssl s_client
-connect 192.168.241.131:5671
-showcerts
With hostname/SNI:
openssl s_client
-connect rabbitmq.example.com:5671
-servername rabbitmq.example.com
Follow RabbitMQ logs:
journalctl -u rabbitmq-server -f
Recent logs:
journalctl -u rabbitmq-server
Last 100 lines:
journalctl -u rabbitmq-server -n 100
Since today:
journalctl -u rabbitmq-server --since today
Check alarms:
rabbitmq-diagnostics alarms
Check node status:
rabbitmq-diagnostics status
Check disk:
df -h
Check memory:
free -h
Check processes:
ps -ef | grep rabbitmq
Create quorum queue:
rabbitmqadmin declare queue
name=sms.quorum
durable=true
arguments='{"x-queue-type":"quorum"}'
List queues and type:
rabbitmqctl list_queues name type
Create DLX:
rabbitmqadmin declare exchange
name=sms.dlx
type=direct
durable=true
Create DLQ:
rabbitmqadmin declare queue
name=sms.dlq
durable=true
Bind DLQ:
rabbitmqadmin declare binding
source=sms.dlx
destination=sms.dlq
destination_type=queue
routing_key=dead
Create main queue with DLX:
rabbitmqadmin declare queue
name=sms.email
durable=true
arguments='{"x-dead-letter-exchange":"sms.dlx","x-dead-letter-routing-key":"dead"}'
Retry queue:
rabbitmqadmin declare queue
name=sms.retry
durable=true
arguments='{"x-message-ttl":30000,"x-dead-letter-exchange":"smsapp","x-dead-letter-routing-key":"sms.email"}'
Meaning:
30,000 ms = 30 seconds TTL.
After expiry: retry queue ↓ main exchange ↓ main queue
List queues:
rabbitmqadmin list queues
List queue details:
rabbitmqadmin list queues
name messages messages_ready messages_unacknowledged
List exchanges:
rabbitmqadmin list exchanges
List bindings:
rabbitmqadmin list bindings
List connections:
rabbitmqadmin list connections
List consumers:
rabbitmqadmin list consumers
rabbitmqctl: Main CLI for RabbitMQ node administration.
Examples:
rabbitmqctl cluster_status rabbitmqctl list_users rabbitmqctl list_queues rabbitmqctl list_connections
rabbitmqadmin: Management HTTP/API-oriented administration tool.
Examples:
rabbitmqadmin declare queue rabbitmqadmin declare exchange rabbitmqadmin declare binding rabbitmqadmin publish
RabbitMQ problem:
rabbitmq-diagnostics ping
rabbitmq-diagnostics status
rabbitmqctl cluster_status
rabbitmq-diagnostics listeners
rabbitmq-diagnostics alarms
rabbitmqctl list_queues
name messages messages_ready messages_unacknowledged
rabbitmqctl list_consumers
rabbitmqctl list_connections
rabbitmqctl list_channels
journalctl -u rabbitmq-server -f
ss -lntp nc -zv
df -h
free -h
rabbitmq-diagnostics ping rabbitmq-diagnostics status rabbitmq-diagnostics listeners rabbitmq-diagnostics alarms
rabbitmqctl cluster_status
rabbitmqctl list_queues rabbitmqctl list_consumers rabbitmqctl list_connections rabbitmqctl list_channels rabbitmqctl list_bindings rabbitmqctl list_exchanges
rabbitmqctl list_users rabbitmqctl list_vhosts rabbitmqctl list_permissions -p /
rabbitmqctl list_policies rabbitmqctl list_parameters
rabbitmqctl federation_status
rabbitmq-plugins list
journalctl -u rabbitmq-server -f ss -lntp df -h free -h
HEALTH: ping → status → listeners → alarms
CLUSTER: cluster_status
QUEUE: list_queues
CONSUMER: list_consumers
CONNECTION: list_connections
CHANNEL: list_channels
ROUTING: list_exchanges → list_bindings
SECURITY: list_users → list_vhosts → list_permissions
FEDERATION: list_parameters → federation_status
NETWORK: ss → nc → epmd
LOGS: journalctl
RESOURCES: alarms → free → df
Linux Quick Study
Linux Kernel — Quick Summary
Kernel: Core component of the OS that acts as a bridge between applications and hardware and manages CPU, memory, filesystems, devices, networking, etc.
Types: Monolithic – major OS functions in kernel space; Microkernel – only essential functions; Hybrid – combination of both; Exokernel – minimal kernel with more direct hardware control.
Kernel Modules: Pieces of kernel code, commonly drivers, that can be dynamically loaded/unloaded without rebooting.
Commands: lsmod → view modules, modprobe module → load, modprobe -r module → remove.
Linux: A monolithic kernel with support for loadable kernel modules.
=========================================================================================================================================
RPO vs RTO — Quick Summary
Term Full Form Simple Meaning Example
RPO Recovery Point Objective How much data can we afford to lose? RPO = 15 min → maximum 15 min of data loss
RTO Recovery Time Objective How quickly must the service be restored? RTO = 2 hours → service must recover within 2 hours
RPO defines the maximum acceptable data loss, while RTO defines the maximum acceptable downtime required to restore the service.
==============================================================================================================
RHEL Boot Process — Quick Summary
BIOS / UEFI
↓
MBR / EFI System Partition
↓
GRUB2
↓
Kernel + initramfs
↓
Kernel initialization
↓
Real root filesystem
↓
systemd (PID 1)
↓
Targets
↓
Services
↓
Login
BIOS/UEFI → Initializes hardware and finds the bootloader.
GRUB2 → Loads the Linux kernel + initramfs.
Kernel → Initializes CPU, memory, devices and other kernel components.
initramfs → Provides drivers/tools to find and mount the real root filesystem.
systemd (PID 1) → Starts userspace services and reaches the configured target.
Target → Brings the system to the required state (multi-user.target / graphical.target).
Finally, services start and the login prompt appears.
===============================================================================
Linux File Types — Quick Summary
- → Regular file → Normal data/config/log files
d → Directory → Contains files/directories
l → Symbolic link → Shortcut/reference to another file
c → Character device → Character/stream-based device
b → Block device → Storage devices like HDD/SSD
s → Socket → Communication between processes
p → Named pipe → Communication between processes using FIFO
================================================================================================
Cron / Crontab — Quick Summary
Cron → Time-based job scheduler used to run commands/scripts automatically.
crond → Background daemon/service that executes cron jobs.
crontab → File containing scheduled jobs.
Basic commands
crontab -l # List jobs
crontab -e # Edit jobs
crontab -r # Remove crontab
crontab -u user -l # View another user's crontab
* * * * * command
│ │ │ │ │
│ │ │ │ └─ Day of week (0-7)
│ │ │ └─── Month (1-12)
│ │ └───── Day of month (1-31)
│ └─────── Hour (0-23)
└───────── Minute (0-59)
0 2 * * * /backup.sh
* → every
, → multiple values
- → range
/ → interval
→ Every 10 minutes.
*/10 * * * * command
Important locations
/var/spool/cron/ → User crontabs
/etc/crontab → System-wide cron
/etc/cron.d/ → System cron jobs
/etc/cron.daily/ → Daily jobs
/etc/cron.weekly/ → Weekly jobs
/etc/cron.monthly/ → Monthly jobs
User crontab:
* * * * * command
/etc/crontab:
* * * * * user command
systemctl status crond
journalctl -u crond
===========================================================================================================================
RHEL UsrMerge — Quick Summary
Modern RHEL uses UsrMerge, where many traditional directories are symlinks to locations under /usr.
/bin → /usr/bin
/sbin → /usr/sbin
/lib → /usr/lib
/lib64 → /usr/lib64
l at the beginning of ls -l → symbolic link
The actual target directories are under /usr.
Old paths are kept as symlinks for backward compatibility.
Example:
/bin/ls
→ follows the symlink → /usr/bin/ls.
Remember: /usr is the main location; /bin, /sbin, /lib, and /lib64 are compatibility links on modern RHEL.
Linux Directory Structure — Quick Summary
/ → Root of entire filesystem
/boot → Kernel, initramfs, GRUB boot files
/etc → System configuration files
/home → Normal users' home directories
/root → Root user's home directory
/bin → Symlink → /usr/bin
/sbin → Symlink → /usr/sbin
/usr → Programs, libraries, shared data
/var → Logs, cache, variable/application data
/tmp → Temporary files
/dev → Device files
/proc → Process & kernel information (virtual)
/sys → Hardware/device & kernel information (virtual)
/run → Runtime data since boot
/opt → Optional/third-party applications
/mnt → Temporary filesystem mount point
/media → Removable media
/srv → Data provided by services
/etc = Configuration | /var = Logs/Data | /boot = Boot | /usr = Programs | /home = Users | /dev = Devices | /proc = Processes | /sys = Hardware | /tmp = Temporary.
=====================================================================================
Linux Links + Inode — Quick Summary
🔹 Inode
Inode is a filesystem data structure that stores file metadata and references to the file's data blocks.
Filename → Inode → File Data
Inode contains:
Permissions
Owner/group
Size
Timestamps
Link count
Data block references
Filename is mapped to the inode by the directory.
🔹 Hard Link
Another filename pointing to the same inode.
file1 ──→ inode 123
file2 ──→ inode 123
Same inode
Same data
Cannot normally cross filesystem
Normally cannot link directories
Original filename deleted → hard link still works
ln file1 file2
Inode → File metadata + data references
Hard link → SAME inode
Soft link → DIFFERENT inode → PATH
======================================================================================================================
/etc/fstab (filesystem table) contains persistent filesystem mount configuration, so filesystems can be mounted automatically during boot.
UUID/device | Mount Point | FS Type | Options | Dump | Pass
UUID=abc123 /data xfs defaults 0 0
UUID/device → Identifies the filesystem.
Mount point → Where it will be mounted, e.g. /data.
FS type → xfs, ext4, etc.
Options → How to mount, e.g. defaults, ro, rw, nofail.
Dump → Legacy backup setting; commonly 0.
Pass → Filesystem check order; commonly 0 for XFS.
fstab = What to mount + Where to mount + Filesystem type + How to mount + Boot/check settings.
============================================================================================================================
NFS — Quick Summary
NFS (Network File System) → Allows a client to access a directory/filesystem from a remote Linux server over the network.
NFS Server → Shares/exports a directory using /etc/exports.
NFS Client → Mounts the exported directory.
dnf install nfs-utils
Server → nfs-utils + nfs-server
Client → nfs-utils
dnf install nfs-utils
Server → nfs-utils + nfs-server
Client → nfs-utils
Firewall
firewall-cmd --permanent --add-service=nfs
firewall-cmd --reload
Server
/etc/exports
/data 192.168.1.20(rw,sync)
exportfs -rav
exportfs -v
mount -t nfs server:/data /mnt/data
/etc/fstab
server:/data /mnt/data nfs defaults,_netdev 0 0
_netdev → tells systemd that the filesystem requires the network.
rw → Read/write
ro → Read-only
sync → Synchronous writes
async → Asynchronous writes
root_squash → Client root mapped to unprivileged user
no_root_squash → Client root not squashed
all_squash → All users mapped to anonymous user
NFSv4 → TCP 2049
NFSv3 → 2049 + RPC services such as rpcbind (111)
Important commands
systemctl status nfs-server
exportfs -v
exportfs -rav
showmount -e server
mount
umount
findmnt
rpcinfo -p server
journalctl -u nfs-server
NFS = Server exports → Client mounts → /etc/exports controls sharing → /etc/fstab makes it persistent → nfs-utils provides NFS tools.
===============================================================================================================================
RHEL PACKAGE MANAGEMENT — INTERVIEW QUICK NOTES
“yum update updates installed packages while generally preserving obsolete packages. yum upgrade also updates packages but can remove obsolete packages when required. For normal maintenance, yum update is commonly used. yum upgrade is useful when you want the installed package set to reflect the current repository state.”
1. RPM vs YUM vs DNF
--------------------
RPM = Low-level package management tool
YUM = Older high-level package manager
DNF = Modern high-level package manager / successor to YUM
RHEL 7 → YUM commonly used
RHEL 8+ → DNF primarily used
RPM works directly with .rpm files.
DNF/YUM work with repositories and resolve dependencies.
IMPORTANT:
Don't say "yum is simply a symlink to dnf".
In modern RHEL, yum is a compatibility interface using DNF
functionality.
2. RPM IMPORTANT COMMANDS
-------------------------
Install:
rpm -ivh package.rpm
Upgrade:
rpm -Uvh package.rpm
Remove:
rpm -e package
List installed:
rpm -qa
Package information:
rpm -qi package
Files installed by package:
rpm -ql package
Find which package owns a file:
rpm -qf /path/file
Show dependencies:
rpm -qR package
Verify package:
rpm -V package
3. DNF IMPORTANT COMMANDS
-------------------------
Install:
dnf install package
Remove:
dnf remove package
Update:
dnf update
Search:
dnf search package
Package information:
dnf info package
Find package providing command/file:
dnf provides '*/command'
Reinstall:
dnf reinstall package
List repositories:
dnf repolist
Clean cache:
dnf clean all
Refresh metadata:
dnf makecache
Transaction history:
dnf history
Undo transaction:
dnf history undo <ID>
4. REPOSITORIES
-------------------------------------------------------------------------------------------
Repository = location containing RPM packages + metadata.
Repository configuration:
/etc/yum.repos.d/*.repo
DNF main configuration:
/etc/dnf/dnf.conf
Important .repo options:
baseurl = repository location
enabled=1 = repository enabled
gpgcheck=1 = verify package signature
5. DEPENDENCY
-------------------------------------------------------------------------------------------
Dependency = another package/library required by software.
RPM:
- Directly installs RPM
- Does not provide DNF-style automatic repository
dependency resolution
DNF:
- Finds package from repository
- Resolves dependencies automatically
6. GPG
-------------------------------------------------------------------------------------------
GPG verifies package signatures.
gpgcheck=1
= Enable package signature verification.
7. RPM DATABASE
-------------------------------------------------------------------------------------------
Location:
/var/lib/rpm/
Contains information about installed RPM packages.
8. RHEL REPOSITORY/SUBSCRIPTION
-------------------------------------------------------------------------------------------
Check subscription:
subscription-manager status
List repositories:
subscription-manager repos --list
Enable repository:
subscription-manager repos --enable=<repo-id>
9. COMMON INTERVIEW SCENARIOS
-------------------------------------------------------------------------------------------
Which package installed a file?
rpm -qf /path/file
Which package provides a command?
dnf provides '*/command'
What files did a package install?
rpm -ql package
Why does DNF say "No match"?
→ Check repository, subscription, metadata and package availability.
Refresh DNF metadata:
dnf clean all
dnf makecache
See recent package changes:
dnf history
⭐ MOST IMPORTANT MEMORY
RPM → Direct .rpm package management
DNF → Repository + dependency management
YUM → Older DNF-style package manager
Repo → Source of RPM packages
GPG → Package signature verification
/var/lib/rpm/ → RPM database
/etc/yum.repos.d/ → Repository files
/etc/dnf/dnf.conf → DNF configuration
RPM → rpm -qa / -qi / -ql / -qf
DNF → install / remove / update / search / provides / history
=============================================================================================
SSHD + PUBLIC KEY AUTHENTICATION
================================
SSH = client
SSHD = SSH server/daemon
Default port = TCP 22
Main config:
/etc/ssh/sshd_config
Important settings:
Port 22
PermitRootLogin no
PasswordAuthentication yes/no
PubkeyAuthentication yes
AllowUsers / AllowGroups
Service:
systemctl status sshd
systemctl restart/reload sshd
Test config:
sshd -t
PUBLIC KEY AUTHENTICATION
=========================
Key pair:
Private key → Client (KEEP SECRET)
Public key → Server
Create:
ssh-keygen -t ed25519
Copy public key:
ssh-copy-id user@server
Server stores public key in:
~/.ssh/authorized_keys
Permissions:
~/.ssh = 700
authorized_keys = 600
private key = 600
Authentication:
Client proves it has private key
↓
Server checks public key in authorized_keys
↓
Login successful
known_hosts = Client stores server host keys
authorized_keys = Server stores user public keys
Troubleshooting:
ssh -vvv user@server
journalctl -u sshd
ss -lntp | grep :22
============================================================================================================================
RSYNC — SHORT INTERVIEW NOTES
============================================================================================================================
Rsync = File/directory synchronization tool.
Used for backup, migration and repeated data transfer.
Basic:
rsync -av /data/ /backup/
Remote:
rsync -av /data/ user@server:/backup/
Important options:
-a = archive/preserve attributes
-v = verbose
-z = compression
-P = progress + partial
-n = dry run
--delete = delete destination files not in source
--exclude = exclude files
Rsync over SSH → TCP 22
Rsync daemon → TCP 873
Daemon config → /etc/rsyncd.conf
IMPORTANT:
/data → copy directory
/data/ → copy contents
RSYNC vs SCP
==============================================================================================================================
SCP = Secure file copy
Rsync = File synchronization
SCP → simple/one-time copy
Rsync → repeated sync, transfers only required changes,
supports --delete, --exclude, --dry-run, etc.
INTERVIEW:
"SCP is for secure copying; rsync is for efficient
synchronization and repeated transfers."
===============================================================================================================================
LVM = Logical Volume Manager
Purpose = Flexible storage management
Architecture:
Disk/Partition → PV → VG → LV → Filesystem → Mount Point
PV = Physical Volume
Physical disk/partition prepared for LVM
Command: pvcreate
VG = Volume Group
Storage pool made from one or more PVs
Command: vgcreate
LV = Logical Volume
Virtual disk created from VG
Command: lvcreate
PE = Physical Extent
LE = Logical Extent
PV → PE
LV → LE
LE maps to PE
Extent size is defined at VG creation:
vgcreate -s 8M vgdata /dev/sdb
Create LV using 100 extents:
lvcreate -l 100 -n lvdata vgdata
Create LV using size:
lvcreate -L 10G -n lvdata vgdata
Common commands:
pvs / pvdisplay
vgs / vgdisplay
lvs / lvdisplay
Add disk:
pvcreate /dev/sdc
vgextend vgdata /dev/sdc
Extend LV:
lvextend -L +10G /dev/vgdata/lvdata
Extend LV + filesystem:
lvextend -r -L +10G /dev/vgdata/lvdata
XFS:
xfs_growfs /data
ext4:
resize2fs /dev/vgdata/lvdata
Snapshot:
lvcreate -L 10G -s -n lvdata_snap /dev/vgdata/lvdata
Important:
LV size ≠ filesystem size
After extending LV, filesystem may also need to be extended.
XFS → can grow, cannot shrink
ext4 → can grow and shrink
LVM Snapshot ≠ Backup
Snapshot = point-in-time view using copy-on-write
Main Advantages of LVM:
1. Online expansion
→ Increase LV/filesystem size while mounted
→ No need to stop the application in many cases
2. Easy storage expansion
→ Add a new disk/PV to an existing VG
→ Then extend the LV
3. Flexible storage allocation
→ Allocate space to LVs as required
→ Don't need fixed partitions for everything
4. Combine multiple disks
→ Multiple PVs can be combined into one VG
5. Snapshots
→ Create point-in-time snapshots of LVs
→ Useful for backups/testing
======================================================================================================================================
Linux permissions control who can access files/directories.
Check:
ls -l
ls -ld /directory
Basic permissions:
r = read = 4
w = write = 2
x = execute = 1
Permission order:
Owner | Group | Others
Example:
-rwxr-xr--
Owner → rwx
Group → r-x
Others → r--
Numeric:
755 = rwxr-xr-x
644 = rw-r--r--
700 = rwx------
600 = rw-------
Commands:
chmod 755 file → change permissions
chown user:group file → change owner + group
chgrp group file → change group
Directory permissions:
r → list contents
w → create/delete/rename files
x → enter/traverse directory
SUID:
4xxx
Executable runs with file owner's effective UID.
Example:
chmod 4755 file
SGID:
2xxx
Executable → runs with file's group
Directory → new files inherit directory's group
Example:
chmod 2775 /shared
Sticky Bit:
1xxx
Users cannot normally delete/rename files belonging to other users
Example:
chmod 1777 /shared
Common example: /tmp
umask:
Controls default permissions for NEW files/directories.
Base:
File → 666
Directory → 777
umask 022:
File → 644
Directory → 755
umask 077:
File → 600
Directory → 700
ACL = Access Control List
Used when owner/group/others are not enough.
Allows specific users/groups to have specific permissions.
Check ACL:
getfacl file
Set ACL:
setfacl -m u:alice:rw file
Remove ACL:
setfacl -x u:alice file
If ls shows '+':
-rw-r-----+
↑
Extended ACL exists
Important difference:
chmod → basic permissions
ACL → granular permissions
umask → default permissions for new objects
SUID → owner privilege
SGID → group privilege / group inheritance
Sticky → restrict deletion in shared directories
======================================================================================================================
IP Address
→ Identifies a host/network interface.
Example: 192.168.1.10
Subnet
→ Defines network + host portion.
Example: 192.168.1.0/24
Gateway
→ Router used to reach other networks.
Check:
ip route
DNS
→ Converts hostname ↔ IP.
Check:
cat /etc/resolv.conf
dig google.com
nslookup google.com
Routing
→ Determines where packets should go.
Check:
ip route
PORTS
→ Identify services on a host.
22 → SSH
53 → DNS
80 → HTTP
443 → HTTPS
123 → NTP
389 → LDAP
636 → LDAPS
445 → SMB
2049 → NFS
3306 → MySQL
5432 → PostgreSQL
8080 → HTTP alternate
TCP
→ Connection-oriented + reliable.
→ SYN → SYN-ACK → ACK
UDP
→ Connectionless + no delivery guarantee.
ss
→ Check listening ports/connections.
ss -lntp
→ Listening TCP + process
ss -lunp
→ Listening UDP + process
ss -tunap
→ All TCP/UDP connections
ss -lntp | grep :22
→ Check SSH port
IP COMMANDS
ip addr
→ Show IP addresses/interfaces
ip link
→ Show interface status
ip route
→ Show routing table
PING
ping 192.168.1.1
→ Test IP connectivity using ICMP
Important:
Ping success does NOT mean application/port is working.
TRACEPATH / TRACEROUTE
tracepath 8.8.8.8
traceroute 8.8.8.8
→ Show network path/hops
DNS
dig google.com
nslookup google.com
→ Test DNS resolution
CURL
curl -v http://server:8080
curl -I https://server
→ Test application/HTTP/HTTPS connectivity
FIREWALL
Check:
firewall-cmd --state
firewall-cmd --get-active-zones
firewall-cmd --list-all
firewall-cmd --list-services
firewall-cmd --list-ports
Allow service:
firewall-cmd --permanent --add-service=http
firewall-cmd --reload
Allow single port:
firewall-cmd --permanent --add-port=8080/tcp
firewall-cmd --reload
Allow UDP:
firewall-cmd --permanent --add-port=53/udp
firewall-cmd --reload
Allow port range:
firewall-cmd --permanent --add-port=8000-8080/tcp
firewall-cmd --reload
Allow UDP range:
firewall-cmd --permanent --add-port=5000-6000/udp
firewall-cmd --reload
Remove port:
firewall-cmd --permanent --remove-port=8080/tcp
firewall-cmd --reload
TCPDUMP
→ Captures actual network packets.
Basic:
tcpdump -i any -n
Specific port:
tcpdump -i any -n port 8080
TCP port:
tcpdump -i any -n tcp port 443
UDP port:
tcpdump -i any -n udp port 53
Specific host:
tcpdump -i any -n host 192.168.1.10
Host + port:
tcpdump -i any -n host 192.168.1.10 and port 443
Source:
tcpdump -i any -n src host 192.168.1.10
Destination:
tcpdump -i any -n dst host 192.168.1.10
Save capture:
tcpdump -i any -n -w capture.pcap
Read capture:
tcpdump -r capture.pcap
Capture 100 packets:
tcpdump -i any -n -c 100
TCP FLAGS
[S] → SYN
[S.] → SYN-ACK
[.] → ACK
[R] → RST
[F.] → FIN-ACK
TCP TROUBLESHOOTING
SYN
→ Client is trying to connect.
SYN → SYN-ACK → ACK
→ TCP connection established.
SYN reaches server but no SYN-ACK
→ Check service, firewall, routing.
SYN → RST
→ Connection refused; often no service listening.
SYN repeatedly, no response
→ Timeout; investigate firewall/network/remote host.
Handshake succeeds but application fails
→ Check application/TLS/HTTP.
NETWORK TROUBLESHOOTING FLOW
ip addr
↓
ip route
↓
ping
↓
ss
↓
firewall-cmd
↓
curl
↓
tcpdump
↓
application logs
tcpdump:
tcpdump -i any -nn
tcpdump -i any -nn port 8080
tcpdump -i any -nn host 10.10.10.20
tcpdump -i any -nn host 10.10.10.20 and port 8080
tcpdump -i any -nn -w capture.pcap
============================================================================================================================
Git Cheat Sheet With Inline Info
# Initialize & Clone
git init # Create a new repository in current folder
git clone <url> # Copy a remote repo locally
# Status & Commit
git status # Show repo state (staged/unstaged files)
git add . # Stage all changes
git commit -m "msg" # Save snapshot with message
git log --oneline # Compact commit history
git diff # Show changes not staged/committed
# Branching
git branch # List branches
git branch <name> # Create new branch
git checkout <name> # Switch branch
git checkout -b <name> # Create + switch branch
git merge <branch> # Merge branch into current
git rebase <branch> # Reapply commits on top of branch
# Remote
git remote -v # Show remotes
git remote add origin <url> # Add remote repo
git push origin main # Push changes to remote main
git pull origin main # Fetch + merge from remote main
git fetch origin # Fetch without merging
# Undo & Reset
git reset <file> # Unstage file
git checkout -- <file> # Discard local changes
git revert <commit> # Undo commit safely (new commit)
git reset --hard <commit> # Reset repo to commit (dangerous)
# Stash
git stash # Save changes temporarily
git stash list # Show stashes
git stash apply # Reapply latest stash
git stash drop # Delete latest stash
Kubernetes Pod Mastery Revision&Nbsp;
# KUBERNETES POD MASTERY — CKA REVISION NOTES
GOAL:
Master Pod YAML, container configuration, volumes, Downward API, probes, resources, security, lifecycle, networking, and troubleshooting.
==================================================
1. BASIC POD STRUCTURE
==================================================
apiVersion: v1
kind: Pod
metadata:
name: mypod
namespace: pod-mastery
labels:
app: demo
spec:
restartPolicy: Always
containers:
- name: nginx
image: nginx:1.27
ports:
- containerPort: 80
IMPORTANT FIELDS:
apiVersion: API version
kind: Resource type
metadata: Name, namespace, labels, annotations
spec: Desired configuration
status: Actual observed state
FIELDS UNDER SPEC:
containers, initContainers, volumes, restartPolicy,
securityContext, nodeSelector, affinity, tolerations,
dnsPolicy, serviceAccountName, terminationGracePeriodSeconds.
FIELDS UNDER CONTAINER:
image, command, args, env, envFrom, ports, resources,
volumeMounts, probes, lifecycle, securityContext,
workingDir, imagePullPolicy.
COMMANDS:
kubectl apply -f pod.yaml
kubectl get pods -n pod-mastery -o wide
kubectl describe pod mypod -n pod-mastery
kubectl get pod mypod -n pod-mastery -o yaml
kubectl delete pod mypod -n pod-mastery
FIELD DISCOVERY:
kubectl explain pod.spec
kubectl explain pod.spec.containers
kubectl explain pod.spec.volumes
kubectl explain pod.spec.containers.env
kubectl explain pod.spec.containers.resources
==================================================
2. LABELS, SELECTORS AND ANNOTATIONS
==================================================
metadata:
labels:
app: payment
tier: backend
annotations:
description: Payment application
kubectl get pods --show-labels
kubectl get pods -l app=payment
kubectl label pod mypod environment=dev
kubectl annotate pod mypod owner=platform-team
Labels identify and select resources.
Annotations store additional metadata.
==================================================
3. COMMAND AND ARGS
==================================================
containers:
- name: worker
image: busybox:1.36
command: ["/bin/sh", "-c"]
args:
- 'echo "Starting"; sleep 3600'
command overrides the image entrypoint.
args overrides the default command arguments.
Shell syntax requires a shell such as /bin/sh -c.
==================================================
4. ENVIRONMENT VARIABLES
==================================================
Direct environment variable:
env:
- name: APP_ENV
value: production
ConfigMap envFrom:
envFrom:
- configMapRef:
name: app-config
One ConfigMap key:
env:
- name: DB_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: database-host
Secret key:
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-secret
key: password
Create resources:
kubectl create configmap app-config --from-literal=APP_ENV=production
kubectl create secret generic db-secret --from-literal=password=example-password
Environment variables do not automatically update when the referenced ConfigMap or Secret changes.
==================================================
5. DOWNWARD API: FIELDREF AND RESOURCEFIELDREF
==================================================
fieldRef exposes Pod metadata or fields.
env:
- name: MY_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MY_NODE
valueFrom:
fieldRef:
fieldPath: spec.nodeName
Other useful fields:
metadata.name
metadata.labels
metadata.annotations
spec.serviceAccountName
resourceFieldRef exposes a container resource request or limit.
containers:
- name: app
image: busybox:1.36
command: ["/bin/sh", "-c", "sleep 3600"]
resources:
limits:
memory: 200Mi
env:
- name: MEMORY_LIMIT
valueFrom:
resourceFieldRef:
containerName: app
resource: limits.memory
Downward API volume:
volumes:
- name: pod-info
downwardAPI:
items:
- path: pod-name
fieldRef:
fieldPath: metadata.name
- path: pod-namespace
fieldRef:
fieldPath: metadata.namespace
Mount it:
volumeMounts:
- name: pod-info
mountPath: /etc/pod-info
readOnly: true
Read:
kubectl exec mypod -- cat /etc/pod-info/pod-name
Remember:
fieldRef = Pod fields.
resourceFieldRef = container resource requests or limits.
==================================================
6. IMAGE SETTINGS, PORTS AND WORKING DIRECTORY
==================================================
containers:
- name: app
image: nginx:1.27
imagePullPolicy: IfNotPresent
workingDir: /tmp
ports:
- name: http
containerPort: 8080
protocol: TCP
Image pull policies:
Always: resolve/pull image when starting.
IfNotPresent: use a cached image when available.
Never: do not pull an image.
containerPort documents the container port; it does not expose the Pod externally.
Private registry example:
spec:
imagePullSecrets:
- name: registry-credentials
containers:
- name: app
image: registry.example.com/team/app:v1
The imagePullSecret must exist in the Pod namespace.
==================================================
7. MULTI-CONTAINER PODS, INIT CONTAINERS AND SIDECARS
==================================================
Containers in a Pod share the network namespace and Pod IP.
They can communicate through localhost.
They do not automatically share their filesystems.
Init container example:
spec:
initContainers:
- name: init
image: busybox:1.36
command: ["/bin/sh", "-c", "echo initialized > /shared/init.txt"]
volumeMounts:
- name: shared-data
mountPath: /shared
containers:
- name: app
image: busybox:1.36
command: ["/bin/sh", "-c", "cat /shared/init.txt; sleep 3600"]
volumeMounts:
- name: shared-data
mountPath: /shared
volumes:
- name: shared-data
emptyDir: {}
An init container must complete successfully before ordinary app containers start.
Native sidecar example:
initContainers:
- name: helper
image: busybox:1.36
restartPolicy: Always
command: ["/bin/sh", "-c", "while true; do sleep 10; done"]
Native sidecars are restartable init containers. Confirm cluster version and feature support.
==================================================
8. POD VOLUMES
==================================================
Volume types:
emptyDir: Temporary Pod-scoped storage.
hostPath: Mount a node filesystem path.
configMap: Configuration files.
secret: Sensitive configuration files.
persistentVolumeClaim: Persistent storage claim.
projected: Combine multiple volume sources.
downwardAPI: Expose Pod metadata as files.
A. EMPTYDIR
volumes:
- name: shared
emptyDir: {}
Mount into each container:
volumeMounts:
- name: shared
mountPath: /shared
Test shared storage:
kubectl exec mypod -c writer -- sh -c 'echo hello > /shared/test.txt'
kubectl exec mypod -c reader -- cat /shared/test.txt
emptyDir survives a container restart within the same Pod but is lost when the Pod is removed.
B. HOSTPATH
volumes:
- name: node-data
hostPath:
path: /data/app
type: DirectoryOrCreate
hostPath depends on the node and has security risks.
C. CONFIGMAP AND SECRET VOLUMES
volumes:
- name: config-files
configMap:
name: app-config
- name: secret-files
secret:
secretName: db-secret
Mount:
volumeMounts:
- name: config-files
mountPath: /etc/app-config
readOnly: true
- name: secret-files
mountPath: /etc/app-secret
readOnly: true
Keys generally become files in the mount directory.
D. SUBPATH
volumeMounts:
- name: config-files
mountPath: /etc/app/settings.conf
subPath: settings.conf
subPath mounts a specific file or directory.
subPath mounts do not receive normal projected ConfigMap/Secret updates.
E. PROJECTED VOLUME
volumes:
- name: combined-config
projected:
sources:
- configMap:
name: app-config
- secret:
name: db-secret
- downwardAPI:
items:
- path: pod-name
fieldRef:
fieldPath: metadata.name
Projected volumes combine multiple sources into one directory.
F. PERSISTENT VOLUME AND PVC
PVC example:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-pvc
namespace: pod-mastery
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
This requires a suitable StorageClass/default provisioner or a matching pre-created PV.
Mount the PVC:
spec:
containers:
- name: app
image: busybox:1.36
command: ["/bin/sh", "-c", "echo persistent > /data/test.txt; sleep 3600"]
volumeMounts:
- name: app-storage
mountPath: /data
volumes:
- name: app-storage
persistentVolumeClaim:
claimName: app-pvc
Check:
kubectl get pv
kubectl get pvc -n pod-mastery
kubectl describe pvc app-pvc -n pod-mastery
emptyDir belongs to the Pod lifetime.
PVC storage can persist beyond an individual Pod, depending on the storage backend and reclaim configuration.
==================================================
9. PROBES
==================================================
Startup probe: Determines whether startup has completed.
Readiness probe: Determines whether the Pod should receive traffic.
Liveness probe: Detects a container that should be restarted.
Example:
containers:
- name: app
image: nginx:1.27
ports:
- containerPort: 80
startupProbe:
httpGet:
path: /
port: 80
periodSeconds: 5
failureThreshold: 30
readinessProbe:
httpGet:
path: /
port: 80
periodSeconds: 5
livenessProbe:
httpGet:
path: /
port: 80
periodSeconds: 10
Probe handlers:
httpGet
tcpSocket
exec
grpc, when supported
Exec probe example:
readinessProbe:
exec:
command: ["/bin/sh", "-c", "test -f /tmp/ready"]
periodSeconds: 5
Readiness failure marks the Pod not ready.
Liveness failure can restart the container.
Startup failure can restart the container after the threshold is reached.
Startup success enables the other probes.
==================================================
10. RESOURCES AND QOS
==================================================
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 200m
memory: 200Mi
100m CPU = 0.1 CPU core.
CPU limits can cause throttling.
Memory limit exceedance can cause OOMKilled.
QoS classes:
Guaranteed: Every container has CPU and memory requests equal to limits.
Burstable: Some resource requests/limits exist but Guaranteed criteria are not met.
BestEffort: No CPU or memory requests or limits for any container.
Check:
kubectl describe pod mypod
kubectl get pod mypod -o jsonpath='{.status.qosClass}{"
"}'
Ephemeral storage:
resources:
requests:
ephemeral-storage: 100Mi
limits:
ephemeral-storage: 500Mi
Node disk pressure can lead to Pod eviction.
==================================================
11. SECURITY CONTEXT
==================================================
Pod-level security:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 2000
Container-level security:
containers:
- name: app
image: nginxinc/nginx-unprivileged
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
Important:
runAsUser: Process UID.
runAsGroup: Primary process GID.
runAsNonRoot: Require non-root execution.
fsGroup: Group-related access for supported volumes.
allowPrivilegeEscalation: Controls privilege escalation.
readOnlyRootFilesystem: Makes the container root filesystem read-only.
capabilities: Controls Linux capabilities.
seccompProfile: Restricts system calls.
Read-only filesystems may require writable temporary volumes.
Volume permissions must be compatible with the chosen UID/GID.
POD SECURITY STANDARDS:
Privileged: Few restrictions.
Baseline: Prevents known privilege-escalation patterns.
Restricted: Stronger security requirements.
POD SECURITY ADMISSION:
enforce: Reject violating Pods.
audit: Record violations.
warn: Warn about violations.
Example:
kubectl label namespace pod-mastery pod-security.kubernetes.io/warn=restricted
==================================================
12. LIFECYCLE AND GRACEFUL TERMINATION
==================================================
containers:
- name: app
image: nginx:1.27
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo started"]
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30
postStart runs after container creation but is not guaranteed to finish before the entrypoint starts.
preStop runs during termination and consumes the grace period.
Processes should handle termination signals properly.
activeDeadlineSeconds limits the total active lifetime of a Pod.
==================================================
13. RESTART POLICIES AND TROUBLESHOOTING
==================================================
restartPolicy values:
Always: Restart exited containers; default.
OnFailure: Restart containers that exit unsuccessfully.
Never: Do not restart exited containers.
Common symptoms:
Pending:
Check scheduling events, resource requests and PVC binding.
ErrImagePull/ImagePullBackOff:
Check image name, registry connectivity and credentials.
CrashLoopBackOff:
Check logs, exit code, command and probes.
CreateContainerConfigError:
Check ConfigMaps, Secrets and referenced keys.
OOMKilled:
Check memory limits and application memory usage.
Running but not Ready:
Check readiness probe and application listener.
Evicted:
Check node memory, disk and ephemeral-storage pressure.
Commands:
kubectl get pods -o wide
kubectl describe pod mypod
kubectl logs mypod -c app
kubectl logs mypod -c app --previous
kubectl exec mypod -c app -- sh
kubectl get events --sort-by=.metadata.creationTimestamp
For multi-container Pods, use -c to select the container.
--previous retrieves logs from the previous container instance.
==================================================
14. DNS, HOSTNAME AND NETWORKING
==================================================
spec:
hostname: app-host
subdomain: app-headless
dnsPolicy: ClusterFirst
DNS policies:
ClusterFirst: Cluster DNS; normal default.
Default: Inherit node DNS settings.
ClusterFirstWithHostNet: Cluster DNS behavior with host networking.
None: Use explicit DNS configuration.
Other fields:
spec:
hostNetwork: false
hostPID: false
hostIPC: false
hostAliases:
- ip: 192.0.2.10
hostnames:
- internal.example
Containers in a Pod share a network namespace, Pod IP and port space.
hostNetwork, hostPID and hostIPC can expose the Pod to host-level resources.
==================================================
15. SERVICE ACCOUNTS
==================================================
spec:
serviceAccountName: app-service-account
automountServiceAccountToken: false
Create:
kubectl create serviceaccount app-service-account -n pod-mastery
kubectl get serviceaccount -n pod-mastery
ServiceAccount provides identity, not permissions by itself.
RBAC controls permissions.
Disable automatic token mounting if the application does not need API access.
==================================================
16. SCHEDULING-RELATED POD FIELDS
==================================================
nodeSelector:
spec:
nodeSelector:
kubernetes.io/hostname: workernode2
Toleration:
spec:
tolerations:
- key: dedicated
operator: Equal
value: app
effect: NoSchedule
Other fields to recognize:
affinity
anti-affinity
topologySpreadConstraints
priorityClassName
runtimeClassName
nodeName
A toleration permits matching taints but does not guarantee scheduling onto a node.
==================================================
17. STATIC PODS AND KUBELET
==================================================
Common kubeadm configuration:
staticPodPath: /etc/kubernetes/manifests
Commands:
systemctl status kubelet
systemctl cat kubelet
cat /var/lib/kubelet/config.yaml
ls -l /etc/kubernetes/manifests/
A Static Pod is managed directly by kubelet from a local manifest.
Many kubeadm control-plane components run as Static Pods.
The path is installation-specific.
==================================================
18. TROUBLESHOOTING WORKFLOW
==================================================
Step 1:
kubectl get pod mypod -o wide
Step 2:
kubectl describe pod mypod
Step 3:
kubectl logs mypod -c app
kubectl logs mypod -c app --previous
Step 4:
kubectl get pod mypod -o yaml
kubectl explain pod.spec.containers
Check:
Field spelling and nesting.
Image and pull credentials.
ConfigMap/Secret names and keys.
Volume definitions and mounts.
Probe configuration.
Resource requests and limits.
Security context and file permissions.
Events and container exit codes.
==================================================
19. ULTIMATEPOD PRACTICE ALREADY COMPLETED
==================================================
Your integrated Pod included:
- nginx-unprivileged main container.
- BusyBox sidecar.
- ConfigMap envFrom.
- ConfigMap volume.
- Shared emptyDir.
- HTTP readiness and liveness probes.
- Exec probes for the sidecar.
- CPU/memory requests and limits.
- Pod securityContext with UID 1000, GID 1000 and fsGroup.
- allowPrivilegeEscalation: false.
Observed result:
Pod Running, 2/2 Ready, Guaranteed QoS.
ConfigMap-mounted APP_ENV file contained production.
Lessons:
Use ls to inspect directories and cat to read files.
Container filesystems are separate unless shared volumes are mounted.
A file under sidecar /tmp is not automatically visible to the main container.
Use a shared volume when containers need to exchange files.
==================================================
20. FINAL COMMAND CHEAT SHEET
==================================================
List Pods:
kubectl get pods -o wide
Inspect Pod:
kubectl describe pod mypod
View YAML:
kubectl get pod mypod -o yaml
Container logs:
kubectl logs mypod -c app
Previous logs:
kubectl logs mypod -c app --previous
Enter container:
kubectl exec -it mypod -c app -- sh
Inspect fields:
kubectl explain pod.spec.containers
ConfigMaps and Secrets:
kubectl get configmap,secret
Storage:
kubectl get pv,pvc
Events:
kubectl get events --sort-by=.metadata.creationTimestamp
Client-side dry run:
kubectl apply --dry-run=client -f pod.yaml
Generate Pod YAML:
kubectl run testpod --image=nginx --dry-run=client -o yaml
==================================================
21. HOW TO MASTER THE 60 PRACTICAL QUESTIONS
==================================================
1. Try writing YAML from memory.
2. If stuck, consult kubectl explain or official Kubernetes documentation.
3. Use AI to understand a field or correct an error, not just copy a solution.
4. Apply the manifest and verify the actual behavior.
5. Inspect events, logs, status and generated YAML.
6. Record recurring mistakes and repeat those questions.
7. Practise combining Downward API, PVCs, projected volumes, init containers, sidecars, probes and security settings.
FINAL CKA RULE:
Understand what each field does, where it belongs, how to find it and how to verify its effect. You do not need to memorize every field perfectly before practising.