RabbitMQ vs Kafka

Why: This is the textbook competing-consumers pattern, and RabbitMQ's queue model is built for exactly it: a worker pulls a job, gets a per-message ack once it's actually done, and a prefetch limit stops one greedy worker from hoarding jobs the others could be doing. If a worker crashes mid-job, the unacked message just goes back in the queue for someone else — no extra plumbing required.

The other one, here: Kafka can technically move the messages, but you'd be rebuilding RabbitMQ's ack/retry/prefetch machinery yourself on top of offset commits, which only tell you "we're past message N," not "job N actually finished." You'd also be capping your worker pool at the partition count for no reason — this workload never needed partitioning in the first place.

Why: RabbitMQ has a first-class pattern for this: Service A publishes with a `reply-to` queue and a `correlation-id`, Service B replies onto that queue, and Service A matches the correlation id back to the waiting caller. It's a direct, one-off message with a direct, one-off reply.

The other one, here: Kafka has no request/reply primitive — you'd fake it with a request topic, a response topic, and your own correlation-id bookkeeping to match replies to callers, on infrastructure whose entire design point is durable streaming, not point-to-point round trips.

Why: This is what exchanges and bindings are for. A topic exchange with bindings like `orders.eu.*` and `orders.*.created` lets the broker itself decide where a message goes, based on a routing key — the routing logic lives in configuration, not in every consumer.

The other one, here: Kafka only routes by partition key. Content-based delivery means every consumer subscribes to the whole topic and filters client-side, or you pre-split the data into many topics up front and hope your routing needs never change shape.

Why: Kafka partitions the topic across brokers to absorb the volume, keys events by user id so each user's events land on the same partition in order, and lets the dashboard, the warehouse loader, and the ML pipeline each run as their own consumer group, reading at their own pace off the same durable log. When the ML pipeline needs to backfill a feature over the last 30 days, that's just resetting its group's offset.

The other one, here: RabbitMQ would need three separate queues bound to the same exchange — three full copies of a multi-million-event-a-day stream — and none of them could be replayed for a fourth consumer that shows up next quarter. The backfill case alone rules it out.

Why: Kafka's log is durable by default and doesn't forget a message once it's read — which is the entire premise of event sourcing: the log is the system of record, and every read model is disposable and rebuildable. Partitioning by aggregate id (an order id, an account id) gives strict per-aggregate ordering, which is the one ordering guarantee event sourcing actually needs.

The other one, here: A RabbitMQ queue deletes a message the instant it's acked. The log can never be the source of truth if it forgets its own history — you'd end up bolting on a separate event store next to RabbitMQ, at which point RabbitMQ is just an unnecessary extra hop in front of the thing that's actually doing the job.

Why: This is Kafka's original use case at LinkedIn, almost unchanged: a high-volume firehose, several independent sinks (Kafka Connect can push to Elasticsearch, S3, a warehouse) each as their own consumer group, and retention long enough that a sink that falls behind — or a brand new one added later — can catch up from history instead of only from "now on."

The other one, here: RabbitMQ fan-out means one queue, and one full copy of the firehose, per sink — expensive at this volume — and a sink that's down for an hour doesn't get to catch up on the hour it missed once its queue's retention or capacity limit is hit. There's no underlying log to rewind.

Why: Partitioning by account id gives every account's transactions strict, in-order processing without forcing the whole ledger through one queue. Long retention (or a compacted topic keyed by account) means the ledger processor, the audit tooling, and a reconciliation job can each read the same authoritative history independently, at any time.

The other one, here: RabbitMQ can give you strict order too — but only per queue, which means one queue per account to avoid serializing every account behind every other one, which isn't operationally realistic at any real scale. And once a transaction is acked and gone, there's no durable history left for an audit to replay.

Why: The device count alone is a partitioning problem — Kafka scales the topic horizontally across brokers and keys by device id, so each device's readings stay ordered without a single queue having to absorb the entire fleet. Alerting, storage, and model training run as independent consumer groups off the same log.

The other one, here: A single RabbitMQ queue tops out well below this volume, and scaling it means sharding queues by hand with no built-in mechanism for it. The fan-out-to-three-independent-systems requirement runs into the same full-copy-per-consumer cost as the log-aggregation case above.

Why: This one is a genuine toss-up, and that's the point: it's simple, low-stakes fan-out with no ordering, no replay and no durability requirement — neither architecture's distinguishing strength is even being tested. RabbitMQ's fanout exchange delivers to every bound queue with almost no configuration. Kafka would work too, at the cost of infrastructure this workload doesn't need.

The trap: The mistake isn't picking either one — it's picking Kafka here out of habit ("it's the scalable one") and inheriting operational weight this traffic never asked for, or picking RabbitMQ and assuming it'll be trivial to bolt on replay/analytics later, when that's precisely the thing it doesn't do. Let the actual requirement decide, not the reputation.