Apache Kafka Core Concepts

Explain Kafka core architecture and key concepts.

What Is Kafka

Apache Kafka is a distributed streaming platform providing high-throughput, low-latency publish/subscribe messaging with durable storage.

Core Concepts

Topic: A logical category for messages, similar to a database table name.

Partition: Topics are split into multiple partitions for parallel read/write, increasing throughput. Messages within a partition are ordered.

Producer: Client that publishes messages to a topic. Can specify a partition key to route related messages to the same partition (ordering guarantee).

Consumer: Client that reads messages from a topic. Consumers track their own offset (read position) and can replay from any offset.

Consumer Group: Multiple consumers forming a group to collectively consume a topic; each partition is read by only one consumer in the group (horizontal scaling).

Broker: Kafka server node. Multiple brokers form a cluster with replication for high availability.

Retention Policy

Messages are retained by time (e.g., 7 days) or size. Consumers can replay historical messages at any time.