Apache Kafka is an open source distributed event streaming platform you can self-host — an Amazon Kinesis and Google Pub/Sub alternative that ingests, stores, and processes high-throughput real-time data streams across your apps and services.

What is Apache Kafka?

Apache Kafka is a distributed event streaming platform that lets systems publish, store, and subscribe to continuous streams of records — events — in real time. A producer writes events to a topic, Kafka persists them as an ordered, replayable log across a cluster of brokers, and any number of consumers read them at their own pace, which decouples your systems and lets the same data feed many uses at once.

Originally built at LinkedIn and now an Apache Software Foundation project, Kafka is written in Java and Scala and is designed to handle trillions of messages a day with millisecond latency. More than 80% of the Fortune 100 run it for data pipelines, analytics, and event-driven applications.

What is Kafka best for?

Kafka is best for high-throughput, durable event streaming — moving large volumes of real-time data between systems and keeping it long enough to re-read. It’s the backbone for log aggregation, metrics pipelines, change-data-capture, real-time analytics, and event-driven microservices, and it shines when many independent consumers need the same stream and the ability to replay history from any point.

What can Kafka do?

Publish and subscribe to streams of events through topics partitioned across a cluster of brokers
Store streams durably with configurable retention — keep data for days or indefinitely, and replay it from any offset
Scale horizontally to thousands of brokers and trillions of messages per day, with latencies as low as 2ms
Process streams in place with the Kafka Streams library — joins, aggregations, windowing, and exactly-once guarantees
Integrate with hundreds of external systems through Kafka Connect’s pre-built source and sink connectors
Replicate partitions across brokers and across regions for fault tolerance and high availability
Run without ZooKeeper using built-in KRaft consensus (Kafka 4.x), simplifying the cluster architecture
Connect from virtually any language via official and community clients — Java, Python, Go, .NET, and more

Where does Kafka fall short?

It’s operationally heavy. Partition planning, consumer-group rebalancing, broker sizing, and monitoring consumer lag take real expertise — teams often budget a meaningful slice of an engineer’s time just to keep a cluster healthy.
It’s overkill for simple task queues. If you only need to hand a job to a worker with flexible routing and per-message acknowledgements, a broker like RabbitMQ is simpler than standing up Kafka.
Its resource appetite is significant. Kafka runs on the JVM and wants Java 17+, ample memory, and fast disks, so it isn’t something you run lightly on a small box.

Is Kafka free?

Yes — Apache Kafka is fully open source under the Apache License 2.0 and free to self-host, so you only pay for the infrastructure it runs on. A commercial ecosystem exists around it — Confluent Cloud and Amazon MSK sell managed Kafka, and Confluent’s platform adds enterprise tooling and support — but the core project is complete and production-ready on its own.

What does Kafka replace?

Kafka is a self-hosted alternative to managed cloud streaming services like Amazon Kinesis and Google Cloud Pub/Sub. It does the same real-time ingest-store-and-distribute job, but you run it on your own infrastructure with configurable retention and no per-shard or per-message billing — the tradeoff being that you operate the cluster yourself.

FAQ

Is Kafka open source? Yes. Apache Kafka is licensed under the Apache License 2.0, an OSI-approved open source license, and is governed by the Apache Software Foundation. You’re free to use, modify, and self-host it without paying.

Can I self-host Kafka for free? Yes. The software is free; you only pay for the servers, storage, and network it runs on. Managed options like Confluent Cloud and Amazon MSK are paid conveniences, not requirements.

Kafka vs RabbitMQ — which should I use? Choose Kafka for high-throughput event streaming, long retention, and replaying large volumes of events to many consumers. Choose RabbitMQ for flexible message routing, task queues, and per-message workflows. They solve overlapping but different problems.

What do I need to run Kafka? Java 17 or newer and a server — or just the official Docker image. Modern Kafka 4.x uses built-in KRaft mode, so you no longer need a separate ZooKeeper cluster. For production, run several brokers with replication for fault tolerance.