Rabbitmq Study
RABBITMQ / RHEL / KUBERNETES / DEVOPS INTERVIEW STUDY
=============================================================
PURPOSE
-------
This is the consolidated study summary of the RabbitMQ topics already learned, followed by the pending topics. Use this as the context for continuing the study later.
=============================================================
PART 1 — RABBITMQ FUNDAMENTALS
=============================================================
1. RabbitMQ is a message broker.
Main flow:
Producer -> Exchange -> Binding -> Queue -> Consumer
2. Producer normally publishes to an exchange, not directly to a queue.
3. Exchange and queue are separate objects.
Creating an exchange does NOT create a queue inside it.
A binding connects an exchange to a queue.
4. Routing key:
- A value supplied by the publisher.
- Direct exchange: exact match.
- Topic exchange: pattern matching through bindings.
- Fanout: routing key is effectively ignored.
- Headers: message headers determine routing.
- Routing key is NOT authentication.
5. Direct exchange:
Exact routing-key match.
Multiple queues can have the same routing key, so one message can be routed to multiple queues.
6. Topic exchange:
- * = exactly one word.
- # = zero or more words.
Example:
binding sms.* matches sms.otp and sms.nonotp
binding sms.# can match sms, sms.otp, sms.otp.high
7. Fanout exchange:
Sends a message to all queues bound to that exchange.
It does NOT broadcast to every queue in RabbitMQ.
8. Headers exchange:
Routes using message headers.
x-match=all -> all matching headers.
x-match=any -> any matching header.
9. Default exchange:
- Name is empty string "".
- Direct exchange.
- Every queue is automatically bound to it using the queue name as routing key.
Example:
exchange="" + routing_key="nonotp" -> queue nonotp
A custom exchange with no binding does NOT fall back to the default exchange.
10. Vhost:
Logical namespace/isolation boundary.
Contains exchanges, queues, bindings and permissions.
/ is the default vhost.
Vhost is not another RabbitMQ server.
=============================================================
PART 2 — QUEUES, DURABILITY AND DELIVERY
=============================================================
11. Durable queue:
Queue definition survives broker restart.
Durable does NOT mean replicated.
12. Persistent message:
Message is marked for durable storage.
For restart durability, normally use durable queue + persistent message.
13. Persistence/durability != HA/replication.
14. Queue types:
- Classic
- Quorum
- Stream
15. Classic queue:
Traditional queue.
Normally hosted on one node and not replicated by default.
Classic queue itself is NOT deprecated.
16. Classic queue mirroring:
Old HA mechanism.
Deprecated in RabbitMQ 3.9.
Removed in RabbitMQ 4.0.
17. Quorum queue:
Replicated queue using Raft consensus.
Usually has one leader and followers.
Each quorum queue has its own Raft group.
There is no single cluster-wide queue leader.
18. Quorum majority:
3 members -> 2/3 majority is enough to continue.
If only 1/3 remains, quorum is lost and safe progress cannot continue.
One-member quorum queue is possible but provides no HA.
19. Quorum leader failure:
Remaining majority elects a new leader.
Application does not need to track the queue leader.
20. Application connection and queue leader are different concepts.
Application can connect to Node3 while the queue leader is Node1.
21. Cluster membership does NOT automatically replicate every queue.
Classic queue data can remain on one node.
22. Classic queue on Node1:
Node2/3 know queue metadata but do not automatically have copies of the messages.
If Node1 restarts, durable/persistent data can normally recover from its disk.
Permanent loss of Node1 can mean loss/unavailability of that classic queue's data.
=============================================================
PART 3 — ACK, NACK, PREFETCH, CONNECTIONS
=============================================================
23. ACK:
Consumer tells RabbitMQ that the delivery was successfully handled.
The message is no longer outstanding for redelivery.
24. NACK:
Consumer explicitly tells RabbitMQ it could not handle the message.
requeue=true -> message can be requeued.
requeue=false -> message is not requeued; with DLX it can be dead-lettered.
25. No ACK is NOT the same as NACK.
If consumer stays connected, message remains unacknowledged.
If channel/connection closes before ACK, message can be requeued/redelivered.
26. RabbitMQ knows a message was delivered because it sent the AMQP delivery.
ACK is the consumer's confirmation of successful handling, not merely confirmation that TCP received a packet.
27. Redelivery:
Same message can be delivered again after requeue or consumer/channel failure.
Applications should be idempotent/deduplicate when duplicate business operations are possible.
RabbitMQ generally provides at-least-once style delivery, not exactly-once business processing.
28. Prefetch:
Controls how many messages can be outstanding/unacknowledged for a consumer under the applicable QoS setting.
Example prefetch=10:
RabbitMQ can have up to about 10 outstanding messages.
If 5 are ACKed, 5 slots become available and RabbitMQ can send 5 more.
Prefetch is NOT messages/second and NOT necessarily a batch size.
RabbitMQ does not wait for all 10 ACKs before replacing freed slots.
29. If prefetch=10 and consumer ACKs nothing:
10 messages remain unacknowledged.
RabbitMQ does not simply send another 10 while the same consumer remains active.
30. If consumer dies before ACK:
Unacknowledged messages can be requeued and delivered again.
31. Connection:
Actual TCP/network connection to RabbitMQ.
32. Channel:
Lightweight logical AMQP session inside a connection.
One connection can have multiple channels.
Connection failure affects its channels.
A channel can fail while the TCP connection remains available for other channels.
33. Publisher confirms:
Broker confirmation to publisher that a publish was accepted according to RabbitMQ's confirmation semantics.
Different from consumer ACK.
=============================================================
PART 4 — TTL, DLX, DLQ AND RETRY
=============================================================
34. TTL:
Time-to-live for messages.
Can be configured at queue level or per message.
No default message TTL.
Effective expiration can be affected by the shorter applicable TTL.
Physical removal may occur when RabbitMQ processes expiration, not necessarily exactly at the expiry timestamp.
35. DLX:
Dead Letter Exchange.
Not automatically enabled.
Can receive messages that are dead-lettered.
36. DLQ:
Dead Letter Queue.
Usually a normal queue bound to a DLX.
RabbitMQ does not automatically consume the DLQ.
37. Dead-letter causes include:
- NACK/reject with requeue=false
- TTL expiration
- Queue length limit
- Other configured dead-letter conditions
38. Typical flow:
Original queue -> DLX -> binding -> DLQ -> separate consumer/process
39. Retry:
Can be implemented using requeue, retry queues with TTL+DLX, or application logic.
requeue=true can cause retry storms.
Temporary failures -> retry.
Permanent/repeated failures -> DLQ.
=============================================================
PART 5 — CLUSTERING, HA AND NETWORK PARTITIONS
=============================================================
40. RabbitMQ cluster:
Multiple RabbitMQ nodes form one logical cluster.
3 nodes are common for production HA but 3 is NOT a technical requirement.
41. Cluster membership:
Nodes use Erlang/RabbitMQ distribution and the Erlang cookie.
Common cookie path:
/var/lib/rabbitmq/.erlang.cookie
42. Common cluster commands:
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@node1
rabbitmqctl start_app
rabbitmqctl cluster_status
43. rabbitmqctl vs rabbitmqadmin:
rabbitmqctl -> node/CLI administration.
rabbitmqadmin -> Management HTTP API based administration.
44. rabbitmqctl reset is destructive.
Do not casually run it in production.
45. Network partition / split brain:
Nodes are alive but cannot communicate.
In a 3-member quorum queue, a 2-node majority can continue while the isolated 1-node minority cannot safely make progress.
Quorum prevents independent minority writes.
46. Application may still have a TCP connection to an isolated minority node.
It may not immediately know that the node is a minority.
Symptoms can be failed operations, missing publisher confirms, timeouts or connection recovery.
Exact timeout/recovery depends on client/application configuration.
47. Publisher confirms help determine whether a publish was accepted.
Missing confirmation does NOT always prove message loss because the broker may have accepted the message while the confirmation was lost.
Blind retry can create duplicates.
Use idempotency/deduplication for business operations such as SMS.
=============================================================
PART 6 — FEDERATION AND SHOVEL
=============================================================
48. Federation:
Connects independent RabbitMQ brokers/clusters.
Does NOT merge them into one cluster.
Useful for multi-DC/region or independent environments.
49. Federated exchange:
Upstream exchange -> federation -> downstream exchange -> normal bindings -> queue -> consumer.
50. Federated queue:
Downstream side can obtain messages from an upstream queue.
It is NOT permanent queue replication and NOT quorum HA.
51. Shovel:
Long-running message transport mechanism.
Consumes from configured source and publishes to configured destination.
Useful for migration, integration and controlled one-way message movement.
Source/destination can be queue/exchange depending configuration.
52. Federation vs Shovel:
Federation -> broker/resource-level ongoing relationship between independent environments.
Shovel -> explicit source-to-destination message transport.
Neither is HA/replication.
=============================================================
PART 7 — SECURITY
=============================================================
53. Authentication:
Answers: Who are you?
RabbitMQ can use local users/passwords or LDAP.
54. Authorization:
Answers: What can you access/do?
Controlled through vhosts and permissions.
55. RabbitMQ permissions:
- configure
- write
- read
56. Useful commands:
rabbitmqctl list_users
rabbitmqctl list_vhosts
rabbitmqctl list_permissions -p /
rabbitmqctl list_user_permissions <user>
57. Example:
rabbitmqctl set_permissions -p /sms smsapp ".*" ".*" ".*"
This grants broad permissions; production should use least privilege.
58. TLS:
5672 -> normal AMQP
5671 -> AMQP over TLS
15672 -> Management HTTP
15671 -> Management HTTPS
59. TLS provides:
Confidentiality
Integrity
Server authentication
60. CA:
Certificate Authority signs certificates.
Java truststore normally contains trusted CA certificates.
61. Truststore:
"Who do I trust?"
Used to verify RabbitMQ server certificate.
62. Keystore:
"Who am I?"
Contains client certificate/private key when client identity is needed, such as mTLS.
63. Normal one-way TLS:
RabbitMQ presents server certificate.
Java client verifies it using truststore.
Client certificate is not normally required.
64. mTLS:
Both sides authenticate using certificates.
RabbitMQ can require client certificates.
Java client uses keystore for client identity and truststore for trusted CA.
mTLS does not replace RabbitMQ authorization.
65. Example RabbitMQ TLS settings:
listeners.ssl.default = 5671
ssl_options.cacertfile = /etc/rabbitmq/certs/ca.pem
ssl_options.certfile = /etc/rabbitmq/certs/server.pem
ssl_options.keyfile = /etc/rabbitmq/certs/server.key
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
66. TLS troubleshooting:
Network -> 5671 -> certificate chain/truststore -> hostname/SAN -> TLS versions/ciphers -> mTLS client cert -> RabbitMQ authentication -> vhost/permissions.
67. LDAP:
RabbitMQ 4.x uses rabbitmq_auth_backend_ldap plugin.
Enable:
rabbitmq-plugins enable rabbitmq_auth_backend_ldap
68. LDAP can authenticate while RabbitMQ internal backend handles authorization:
auth_backends.1.authn = ldap
auth_backends.1.authz = internal
69. LDAP authorization can use:
vhost_access_query
resource_access_query
topic_access_query
tag_queries
Exact advanced.config syntax should be verified against the target RabbitMQ version before production.
=============================================================
PART 8 — BACKUP, RESTORE AND MIGRATION
=============================================================
70. Definitions backup:
rabbitmqctl export_definitions /backup/rabbitmq-definitions.json
71. Restore:
rabbitmqctl import_definitions /backup/rabbitmq-definitions.json
72. Definitions include:
vhosts, users, permissions, exchanges, queues, bindings, policies, parameters, etc.
73. Definitions backup does NOT contain queued message bodies.
74. There is no general native "dump one queue's messages and restore it" command.
Message-level export can be done with application logic, Management API get/requeue workflows, shovel, or consume/re-publish strategies depending requirements.
75. Do not casually copy a live /var/lib/rabbitmq directory as a consistent backup.
For message-store/data backup, follow the RabbitMQ-supported procedure and normally stop the node for a consistent filesystem-level backup.
76. Blue-Green/fresh-cluster migration:
- Move application to DR
- Stop/quiesce producers
- Verify production queues are drained
- Export definitions
- Backup configuration/certs
- Stop old cluster
- Provision clean RabbitMQ 4.x + compatible Erlang
- Build new cluster
- Import definitions
- Validate
- Switch application back
77. This is a fresh-cluster/Blue-Green migration, NOT a rolling upgrade.
78. Rolling upgrade:
Only use a source->target version path explicitly supported by RabbitMQ.
Mixed versions may temporarily coexist when supported.
Upgrade one node, verify, continue.
Feature flags should be enabled only according to the supported upgrade process.
=============================================================
PART 9 — START/STOP, HEALTH CHECKS AND DATA DIRECTORY
=============================================================
79. Useful commands:
systemctl status rabbitmq-server
systemctl stop rabbitmq-server
systemctl restart rabbitmq-server
rabbitmqctl stop
rabbitmq-diagnostics status
rabbitmq-diagnostics ping
rabbitmq-diagnostics alarms
rabbitmq-diagnostics listeners
rabbitmqctl cluster_status
80. Queue/consumer checks:
rabbitmqctl list_queues name type messages messages_ready messages_unacknowledged
rabbitmqctl list_connections
rabbitmqctl list_channels
rabbitmqctl list_consumers
81. RabbitMQ data directory commonly:
/var/lib/rabbitmq
82. Custom data directory can be configured using:
RABBITMQ_MNESIA_BASE=/app/rabbitmq
usually in /etc/rabbitmq/rabbitmq-env.conf
83. User's lab:
NODENAME=rabbit@node8
RABBITMQ_MNESIA_BASE=/app/rabbitmq
84. Active Mnesia directory can be checked:
rabbitmqctl eval 'rabbit_mnesia:dir().'
85. Important:
Stop RabbitMQ before moving/copying node data.
Preserve ownership/permissions.
Do not blindly merge Mnesia databases.
Do not delete the old directory until the new setup is validated.
Custom paths on RHEL/Rocky may require SELinux context handling.
Never share one RabbitMQ data directory between multiple RabbitMQ nodes.
=============================================================
PART 10 — MEMORY, DISK, CPU AND PERFORMANCE
=============================================================
86. RabbitMQ uses both memory and disk.
Do not simplify this to "messages are always in RAM and then move to disk."
Internal storage behavior is more sophisticated.
87. Memory alarm:
RabbitMQ has a memory high-watermark/protection mechanism.
When memory pressure reaches the threshold, RabbitMQ can raise a memory alarm and apply publisher flow control.
88. Commands:
rabbitmq-diagnostics alarms
rabbitmq-diagnostics memory_breakdown
free -h
top
89. free -h is OS-wide, not RabbitMQ-specific.
It includes filesystem cache, kernel memory and other processes.
90. Disk alarm:
RabbitMQ protects itself when available disk space becomes too low.
Publishers can be blocked/slowed.
91. Commands:
df -h
df -h /app/rabbitmq
rabbitmq-diagnostics alarms
92. Disk capacity vs disk performance:
df -h -> how much space is available
iostat -xz 1 -> disk I/O/performance
93. CPU:
top
mpstat -P ALL 1
94. Network:
sar -n DEV 1
ip -s link
tcpdump
95. Performance depends on:
CPU
memory
disk I/O
network
message size
message rate
persistence
publisher confirms
queue type
quorum replication
consumers
prefetch
downstream dependencies
96. Producer rate vs consumer rate:
If producer > consumer, backlog grows.
backlog growth approximately = producer rate - consumer rate.
97. Example:
Producer 10,000 msg/sec
Consumer 7,000 msg/sec
Backlog grows about 3,000 msg/sec.
98. Message size matters:
200 KB × 5,000 msg/sec ≈ 1 GB/sec raw payload.
This is before protocol/replication/storage overhead.
99. Connection capacity:
No universal connections/sec limit.
Concurrent connections and connection-establishment rate are different.
Reuse long-lived connections.
OS file descriptors, CPU, RAM, TLS and workload affect capacity.
100. User's example hardware:
8 GB RAM, 500 GB disk, 8 CPU cores, 10 Gbps NIC.
Previous rough planning estimate: 5k–10k concurrent connections per node as an initial planning range, not a RabbitMQ limit; benchmark actual workload.
101. 80 million messages:
Storage depends on average message size.
80M × 200 KB ≈ 16 TB raw payload.
Message count alone is insufficient for capacity planning.
=============================================================
PART 11 — QUEUE TROUBLESHOOTING AND PERFORMANCE
=============================================================
102. messages_ready:
Messages waiting to be delivered.
103. messages_unacknowledged:
Messages delivered to consumers but not yet ACKed.
104. High ready + low unacked:
Consumers absent, slow, disconnected or unable to keep up.
105. Low ready + high unacked:
Consumer has messages but is slow/stuck/not ACKing.
106. Both zero:
Queue is empty; could be healthy or producer may not be publishing.
107. Consumer throughput:
Depends on number of consumers and processing time, but downstream DB/API/workload often becomes the real bottleneck.
108. Example:
100 ms/message ≈ 10 msg/sec/consumer.
10 consumers ≈ 100 msg/sec in a simple idealized case.
109. High prefetch:
Can produce many unacknowledged messages, memory pressure and uneven work distribution.
Do not blindly increase it.
110. Low prefetch:
Can reduce pipeline utilization if consumer processing has latency.
111. Good tuning method:
Measure consumer processing time, ready/unacked counts and throughput.
Adjust prefetc