What is the need of zookeeper in kafka Can I use kafka without zookeeper

Question

Hi,

In kafka configuration we have to first configure zookeeper. Is their anyway, so that we can use kafka without zookeeper?

Thank You

Answer 1

Hi@akhtar,

You can not use kafka without zookeeper. The reasons behind this is given below.

Mainly zookeeper is used to manage all the brokers.  These brokers are responsible for maintaining the leader/follower relationship for all the partitions in kafka cluster. Whenever problem occurs and nodes shut down, it is the controller from the broker that tells other replicas to become partition leaders to replace the partition leaders on the node that is going away. So zookeeper is  used to elect one controller from the brokers.

Zookeeper also manages the status of the brokers, which broker is alive or dead.

Zookeeper also manages all the topics configuration, which topic contains which partitions etc.

Hope this will clear your doubt.

To know more about Kafka, I would recommend you to enroll with Kafka training online today.

Thanks.

Comments

There is a plan in the works to remove the zookeeper requirement.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum

Answer 2

At present, There is no any alternative for zookeeper in Kafka. Kafka uses ZooKeeper to manage the cluster. ZooKeeper is used to coordinate the brokers/cluster topology. ZooKeeper is a consistent file system for configuration information. ZooKeeper gets used for leadership election for Broker Topic Partition Leaders. 

Finally, I want to say ConfuentInc planning to launch new version Kafka without zookeeper. So hopefully next version will give relax to start many things. 

Answer 3

I just would like to complete the answers. I have read the documentation of Kafka and in the QuickStart section (see: http://kafka.apache.org/documentation/#quickstart) and this is written:

# Start the ZooKeeper service
# Note: Soon, ZooKeeper will no longer be required by Apache Kafka.
$ bin/zookeeper-server-start.sh config/zookeeper.properties 

(Currently latest version is: kafka_2.13-2.6.0)