- In 2020, the Apache Kafka project started work to remove the Zookeeper dependency (KIP-500)
- Zookeeper shows scaling issues when Kafka clusters have > 100,000 partitions
By removing Zookeeper, Apache Kafka can:
- Scale to millions of partitions, and become easier to maintain and set up
- Improve stability, making it easier to monitor, support, and administer
- Use a single security model for the whole system
- Have a single process to start with Kafka
- Achieve faster controller shutdown and recovery time
- Kafka 3.x now implements the Raft protocol (KRaft) to replace Zookeeper
- Production-ready since Kafka 3.3.1 (KIP-833)
