Apache Kafka

Name: Apache Kafka
Author: Apache Software Foundation

In-depth entry

by Apache Software Foundation · Platform · also known as Kafka

Distributed event streaming platform with durable, partitioned, replayable logs, Kafka Connect connectors and Kafka Streams processing

Apache-2.0 licenseOpen sourceFreeSelf-hostedCloudSelf-hostable

Visit official site Compare

Link verified Nov 2, 2026

Overview

Apache Kafka is an open-source distributed event streaming platform maintained by the Apache Software Foundation under Apache 2.0. Producers append records to topics that are split into partitions and replicated across brokers; consumers read partitions at their own offsets, so the same stream can feed many independent applications and be replayed from retained history. Since the 4.0 release the KRaft consensus protocol replaces ZooKeeper for cluster metadata, reducing the components to operate. Kafka Connect runs source and sink connectors for databases, object storage, search engines and SaaS systems, Kafka Streams is a Java library for stateful stream processing, and the ecosystem adds schema registries, Flink or ksqlDB processing and clients for Java, Python, Go, JavaScript, .NET and Rust, many based on librdkafka.

Kafka is the backbone for event-driven architectures, change-data capture, log and metrics pipelines, and real-time feature or market-data feeds. Managed services (Confluent Cloud, Amazon MSK, Aiven, Redpanda's compatible platform) remove most operations, and Confluent develops much of the ecosystem commercially.

It is a heavyweight choice for small systems: a JVM-based cluster with careful partition, retention and consumer-group design is needed, ordering is per partition only, there is no per-message acknowledgement or priority queueing in the style of RabbitMQ, and latency is tuned for throughput rather than sub-millisecond delivery. For a single application that only needs a job queue, RabbitMQ, NATS, pg-boss or BullMQ are lighter alternatives.

Capabilities

Partitioned, replicated, append-only topic logs with configurable retention
Consumer groups with offsets and replay
KRaft metadata consensus without ZooKeeper
Kafka Connect source and sink connectors
Kafka Streams library for stateful processing
Exactly-once semantics and transactions
Tiered storage and compaction
Clients in Java, Python, Go, JavaScript, .NET, Rust and more

Typical workflows

Change-data capture from PostgreSQL into a lake or search index
Event bus between microservices
Ingesting market data or telemetry for real-time analytics
Log pipeline feeding ClickHouse or Elasticsearch

Strengths

Durable replayable history decouples producers and consumers
Horizontal scalability and high throughput
Large connector and processing ecosystem
Multiple managed offerings and compatible alternatives

Limitations

Cluster operations and JVM tuning are demanding
Ordering guaranteed only within a partition
No message priorities or per-message routing like AMQP brokers
Overkill for simple job queues

Prerequisites

JVM and several broker nodes for production, or a managed service

Who it is for

Platform and data engineers
Backend teams building event-driven systems

Alternatives & comparison

RabbitMQ is a message broker with flexible routing, acknowledgements and priorities but, outside its Streams feature, deletes messages once consumed, whereas Kafka keeps a replayable log. NATS offers a lightweight core pub/sub with JetStream persistence and far simpler operations at smaller scale. Redis Streams, pg-boss and BullMQ are application-level queues that reuse Redis or PostgreSQL and fit single-application job processing rather than organization-wide streaming.

BullMQ
Redis-backed job and message queue for Node.js (with a Python port) offering workers, flows, rate limiting, repeatable jobs and a commercial Pro tier
MixedOpen source
NATS
Lightweight Go messaging system with core pub/sub and request-reply plus JetStream persistence, key-value and object stores; a CNCF project
Open sourceOpen source
pg-boss
Node.js job queue that uses PostgreSQL tables and SKIP LOCKED for reliable background jobs, scheduling and retries without a separate broker
Open sourceOpen source
RabbitMQ
Open-source message broker in Erlang supporting AMQP 0-9-1 and 1.0, MQTT, STOMP and Streams, with routing, acknowledgements and quorum queues
Open sourceOpen source
Redis
In-memory key-value data store used as cache, message broker and database, with data structures, Pub/Sub, Streams and, since Redis 8, built-in JSON, search and time-series modules
MixedOpen source

Compare Apache Kafka with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

License
Kafka LICENSE (Apache 2.0)
https://github.com/apache/kafka/blob/trunk/LICENSE
Supports: license_spdx, source_model
checked Nov 1, 2026
Official documentation
Kafka documentation
https://kafka.apache.org/documentation
Supports: capabilities, languages
checked Nov 1, 2026
Official site
Apache Kafka
https://kafka.apache.org/
Supports: name, summary
checked Nov 1, 2026
Repository
apache/kafka
https://github.com/apache/kafka
Supports: repo_url
checked Nov 1, 2026

Confidence: High
Reviewed: Nov 1, 2026
Links: Verified · Nov 2, 2026