Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming applications. It allows publishing, storing, and processing large volumes of events in a fault-tolerant and scalable manner.
| Component | Description |
|---|---|
| Producer | Sends messages/events to Kafka topics. |
| Consumer | Reads messages from Kafka topics. |
| Topic | Logical channel where messages are published. |
| Broker | Kafka server responsible for storing and serving data. |
| Partition | Unit of parallelism and scalability within a topic. |
| ZooKeeper / KRaft | Manages metadata, broker coordination and leader election. |
A Kafka Topic is a logical channel where producers publish messages and consumers subscribe to receive them. Topics can contain multiple partitions for scalability.
A partition is an ordered, immutable sequence of records. Multiple partitions allow Kafka to process messages in parallel.
A Kafka Broker is a server responsible for storing messages and serving producer and consumer requests. A Kafka cluster consists of one or more brokers.
ZooKeeper manages broker metadata, leader election, configuration management and cluster coordination.
Modern Kafka versions are moving towards KRaft mode, which eliminates the ZooKeeper dependency.
A Consumer Group is a collection of consumers working together to process records from a topic. Kafka guarantees that each partition is consumed by only one consumer within a group.
Kafka replicates partitions across multiple brokers for fault tolerance. Each partition has:
If the leader fails, Kafka automatically promotes a follower replica.
Kafka Streams is a Java client library used to build real-time stream processing applications and microservices.
| Policy | Description |
|---|---|
| Time-Based | Retain messages for a fixed duration (e.g., 7 days). |
| Size-Based | Retain messages until storage limit is reached. |
| Compaction | Retain only the latest record for each key. |
Food Delivery Application
Kafka ensures all systems receive the required events without directly communicating with each other.
Spring WebSocket provides support for real-time bidirectional communication between client and server using WebSocket protocol.
| Component | Description |
|---|---|
| WebSocket | Full-duplex communication protocol. |
| STOMP | Messaging protocol on top of WebSocket. |
| SockJS | Fallback support when WebSocket is unavailable. |
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-websocket</artifactId>
</dependency>
Client | v /app/send | v @MessageMapping | v /topic/messages | v Subscribed Clients
| Annotation | Purpose |
|---|---|
| @EnableWebSocketMessageBroker | Enables WebSocket messaging. |
| @MessageMapping | Maps client messages. |
| @SendTo | Broadcasts response to subscribers. |
| @Controller | Handles WebSocket requests. |
| Feature | Kafka | RabbitMQ |
|---|---|---|
| Model | Publish-Subscribe | Queue-Based |
| Throughput | Very High | Moderate |
| Retention | Configurable | Deleted After Ack |
| Ordering | Within Partition | Limited |
| Protocol | Kafka Protocol | AMQP |
| Best For | Event Streaming | Task Queues |
| Feature | Kafka | JMS | Azure Service Bus |
|---|---|---|---|
| Messaging Model | Event Streaming | Queue & Topic | Queue & Topic |
| Scalability | Very High | Medium | High |
| Cloud Native | No | No | Yes |
| Throughput | Very High | Medium | High |
| Retention | Configurable | Temporary | Configurable |
| Protocol | Kafka Protocol | JMS | AMQP |
| Best Use Case | Streaming | Enterprise Messaging | Cloud Messaging |
Apache Kafka is a distributed event-streaming platform used for high-throughput messaging.
Producer --> Topic --> Broker --> Consumer
| Topic | Partition |
|---|---|
| Logical message category | Physical subdivision of topic |
| Stores events | Provides scalability & parallelism |
Topic: Orders +- Partition-0 +- Partition-1 +- Partition-2
Multiple consumers working together to consume messages from a topic.
Topic +- Consumer-1 +- Consumer-2 +- Consumer-3
A replica broker becomes leader and continues serving requests.
Broker-1 (Leader) Down
?
Broker-2 becomes Leader
Copies topic partitions across multiple brokers for fault tolerance.
Partition-0 +- Leader +- Replica-1 +- Replica-2
Replicas fully synchronized with the leader partition.
ISR = {Leader, Replica-1, Replica-2}
| Kafka | RabbitMQ |
|---|---|
| Event Streaming Platform | Message Broker |
| High Throughput | Low Latency Messaging |
| Stores Events | Deletes After Processing |
| Partition Based | Queue Based |
KRaft (Kafka Raft Metadata Mode) removes ZooKeeper dependency.
Old: Kafka + ZooKeeper New: Kafka KRaft Mode
Defines how long Kafka keeps messages.
retention.ms=604800000 (7 Days)
Spring WebSocket enables real-time bidirectional communication.
Used In:
Client ? Server Real-Time Communication
| WebSocket | REST API |
|---|---|
| Persistent Connection | Request/Response |
| Bidirectional | Unidirectional |
| Real-Time Updates | Polling Required |
| Low Latency | Higher Overhead |
STOMP (Simple Text Oriented Messaging Protocol) provides messaging semantics over WebSocket.
Client Subscribe:
/topic/messages
Server Send:
@SendTo("/topic/messages")
| Question | One-Line Answer |
|---|---|
| Kafka | Distributed event-streaming platform for real-time data processing. |
| Topic | Logical channel where messages are stored. |
| Partition | Physical subdivision of a topic for scalability. |
| Consumer Group | Group of consumers sharing message processing workload. |
| Broker | Kafka server responsible for storing and serving data. |
| Broker Failure | Replica automatically becomes leader. |
| Replication | Copies data across brokers for fault tolerance. |
| ISR | Replicas synchronized with leader partition. |
| Leader Partition | Handles all read and write requests. |
| Follower Replica | Copies data from leader partition. |
| Kafka vs RabbitMQ | Kafka is streaming; RabbitMQ is messaging. |
| KRaft Mode | Kafka metadata management without ZooKeeper. |
| Retention Policy | Determines how long Kafka stores messages. |
| Log Compaction | Retains latest value per message key. |
| Offset | Unique identifier of a record within a partition. |
| Spring WebSocket | Framework for real-time bidirectional communication. |
| WebSocket | Persistent full-duplex communication protocol. |
| REST API | Request-response communication over HTTP. |
| STOMP | Messaging protocol used over WebSocket. |
| @SendTo | Sends messages to subscribed WebSocket clients. |
| /topic | Broadcasts messages to multiple subscribers. |
| /queue | Point-to-point messaging destination. |
| WebSocket Use Cases | Chat, notifications, stock updates, gaming. |