RabbitMQ vs Kafka

What is Rabbit?
What is Kafka?
Why Rabbit MQ?
Why Kafka?
When to use RabbitMQ over Kafka?
When to use Kafka over RabbitMQ?
Challenges of RabbitMQ
Challenges of Kafka

What is Rabbit?

General-purpose message broker, based around message queues, designed with a smart broker / passive consumer model

What is Kafka?

Distributed & partitioned commit log with messaging semantics. Distributed real-time streaming platform.

Q? How log a log base system that is real-time streaming
Q? log base system are slow but real-time systems are supposed to be fast how does it work

Why Rabiit MQ?

Client Libraries
- Java(4), Spring(3), .Net(6), Ruby(7), Python(4), PHP(7), JavaScript & Node(4), Rust(2), Object-C Scala(1), Other JVM(11), C & C++(4), Go(3)
Plugins
- Tier 1(19) Community: Routing, Auth, Mgmt, Clustering, Logging, Queues, Protocols
Message Dash board
Powerful routing capabilites it has
Easy to Scale by adding/removing competing consumers on a single queue
Can be configured for consistency, high availabilty, low latency, high throughput
Cluster rolling upgrade via feature flags
Easy to get started
Supports strict ordering
Wider use case: event driven microservices, RPC, ETL (with SCDF), enterprise message bus, pub-sub message, real-time analytics (with Reactor and RabbitMQ Reactive API).

Why Kafka?

Very high throughput with data guarantees
Massive Scale (It can scale massively)
Ability to replay events (It is a log-based system)
De-facto standard for streaming platform
Ability to plug-n-play consumer groups on a topic
Support strict ordering [with a partition]
Replace complex data architectures
Broad ecosystem
Wider use cases: pub-sub messaging, events driven microservices, logs store, streaming, event sourcing, CDC, enterprise data pipelines.

When to use RabbitMQ over Kafka?

If you don’t have specific Kafka requirements, then RabbitMQ gives greater flexibility, can meet high throughput and real-time event-processing needs and has lower cost of operations

for example) event replay
operational costs there’s both the hardware as well as the operational complexity

Evolving application requirements

Decoupled producer and consumers (using exchanges)

Consumers independently bring their own queue that binds to exchanges

Consuming applications don’t need to process message that aren’t relevant

When to use Kafka over RabbitMQ?

Steaming platform

If you need streaming platform If you need have streaming requirements, kafka is usually becoming the choice

Extremely high throughput

It can scale it can have parallelism because of the partitions. It can scale really well

Joining multiple streams or streams and tables to enrich the data.

Scale (typically beyond 5 brokers)

Replay

the individual consumer actually can replay.
Traditional message queue where when you read the message the message is gone from queue

Challenges of RabbitMQ

Operational Complexity in resolving network partitions

Relative operational complexity is lower than it is with Kafka

Queues are single-threaded

Which means that throughput is limited
It can’t scale beyond the compute capacity of the broker

Scaling brokers

a rabbit cluster it is really optimal to do a three-node cluster. beyond three brokers for example five clusters, you can do that it becomes a little bit complicated. If you scale beyond a five node cluster it really you have some performance impacts.

No events replay

Does not natively support stateful streaming use cases (but can do so with Reactor + RabbitMQ Reactive API with an external store such as Redis)

Challenges of Kafka

No free lunch – operational complexity

Requires a separate Zookeeper cluster

Requires meticulous planning to select partition count

Storage management overheads

Careful coordination needed between teams writing consumer groups and producers

No out of box management & monitoring console

Streaming API support restricted mainly to Java

If you want to get the latest versions of the streaming API, Java is the best choice

No out of box solution for upgrade.

confluent offer some solution but it is not part of the open-source.