Design a Distributed Message Queue
You're asked to design a distributed message queue like Kafka, SQS, or RabbitMQ. Producers publish messages to named topics, and consumers read them asynchronously. The system must handle hundreds of thousands of messages per second without losing acknowledged data, survive individual machine failures, and let consumers scale independently from producers.
In this prompt, the queue itself is the product. Let's explain how topics, partitions, consumer offsets, replication, and failure recovery work rather than treating the queue as an existing dependency.
Clarify the requirements
- Queue or pub/sub? In a queue, one consumer processes each message. In pub/sub, every subscriber receives a copy. Consumer groups can support both models in one system.
- What delivery guarantee? Choose among at-most-once, at-least-once, and exactly-once. Use at-least-once by default, then explain why a crash between processing and acknowledgement prevents exactly-once delivery.
- Does ordering matter? Global ordering across a topic is expensive. Per-key ordering scales much better and matches what most consumers need.
- How long are messages retained? Delete-on-consume and retain-for-N-days are different designs. Retention enables replay, which is a major feature.
- How large are messages? Kilobytes changes nothing; megabytes means storing payloads elsewhere and passing references.
Back-of-envelope numbers
- Ingest:
- Retained bytes over 7 days:
- With 3× replication: , a large but ordinary storage footprint
Commodity disks can handle half a gigabyte per second of sustained sequential writes. The append-only workload avoids the random writes that would make the same throughput much harder to achieve.
High-level architecture
- Brokers. Each broker leads some partitions and appends incoming messages to their logs on disk.
- Followers. Keep replicas of partitions on other brokers current and take over when a leader fails.
- Consumer group. A set of consumers that jointly read a topic, with each partition assigned to exactly one member.
- Metadata store. The map of which broker leads which partition, held in a strongly consistent store like ZooKeeper or etcd.
Deep dive 1: the partitioned log
A topic is a logical stream split physically into partitions. Each partition is an append-only log file. The broker writes messages sequentially, assigns each one a monotonically increasing offset, and never modifies entries in place.
This structure supports high throughput because appending to a file and reading forward from an offset are efficient disk operations. The broker does not update an index, lock rows, or perform random seeks for each message.
Partitions scale independently across brokers, so adding partitions and brokers increases throughput. The partition key determines where a message lands: hash the key and take the result modulo the partition count. Messages with the same key reach the same partition and retain their order.
Deep dive 2: ordering, and why it's per-key
Global ordering requires every message to pass through one sequence-number authority, limiting throughput to that single point. Use ordering within a partition instead.
With ordering within a partition, consumers read messages in the order the broker wrote them. All events for one key, such as an order, user, or document, arrive in order because they share a partition. Events for different keys can still run in parallel.
Connect the partition key to both ordering and load distribution. A low-cardinality key such as country creates hot partitions, while random partitioning loses per-key order.
Deep dive 3: consumers, offsets, and delivery semantics
Consumers pull messages and track an offset for their position in each partition. A slow consumer falls behind instead of being overwhelmed, while the messages remain safely on disk.
Within a consumer group, assign each partition to one member so only one consumer processes each message. Two separate groups reading the same topic each receive every message, which provides pub/sub behavior with the same mechanism.
Offset commits are where delivery semantics are decided:
- Commit before processing gives at-most-once. If the consumer crashes after committing, that message is never processed.
- Commit after processing gives at-least-once. If the consumer crashes after processing but before committing, the message is redelivered.
Use at-least-once delivery by default. Wherever the consumer commits its offset, a crash can occur between processing and acknowledgement. Exactly-once processing is still possible when consumers are idempotent and can safely handle a redelivery.
Deep dive 4: replication and broker failure
Each partition has a leader and some number of followers on other brokers, which is leader-based replication applied per partition. Producers write to the leader; followers pull from it and append the same messages in the same order.
Choose when to acknowledge a write. Acknowledging after the leader receives it is fast but can lose data if the leader dies before replication. Waiting for a quorum of in-sync replicas adds latency but survives one broker loss. Make this choice per topic because metrics and payment events may need different durability guarantees.
When a broker dies, promote a follower. Track which replicas are in sync because promoting a lagging follower would lose messages it had not received. Let the strongly consistent metadata store decide the new leader so two brokers never lead the same partition simultaneously.
Deep dive 5: retention and replay
Because consumers track their own offsets, the broker does not need to delete a message after it is read. Retaining messages lets a consumer reset its offset after a bug and reprocess history. A new consumer can also start from the beginning of the retained log.
Delete whole log segments when they age out instead of scanning individual messages. An alternative policy, compaction, retains only the newest message for each key and turns the topic into a changelog of current state.
Common pitfalls
- Promising global ordering. It requires a single writer and doesn't scale.
- Claiming exactly-once delivery. Explain the gap between processing and acknowledging instead.
- Pushing to consumers. Push overwhelms slow consumers; pull lets them fall behind harmlessly.
- Ignoring in-sync replica tracking. Promoting a lagging follower silently loses acknowledged messages.
- Choosing a low-cardinality partition key. Creates hot partitions that cap throughput regardless of how many brokers you add.
Leveling signals
Related lessons
Build a distributed key-value store with replication and tunable consistency.
Deliver events to customer HTTP endpoints that may be slow, failing, or offline.