Setting Up kafka on multiple servers

Question

To install kafka and zookeeper. Everything works fine, but I'm setting up kafka and zookeeper on 2 servers.

The kafka/config/server.properties are:

• Server 1: broker.id = 0
• Server 1: zookeeper.connect = localhost:2181,99.99.99.91:2181
• Server 2: broker.id = 1
• Server 2: zookeeper.connect = localhost:2181,99.99.99.92:2181

I'm confused about the following:

• When I publish a topic, does it go to both Instances, or just the server it's loaded on?
• In order to use multiple servers like this, would I be required to use something like HAProxy with say 3 servers?
• Is there anything important I am missing with using 3 servers?

Answer 1

I'm assuming that instances you mentioned are kafka servers.

Now, to answer your questions:

#1: New created topics info is stored on zookeeper and some of the key information will be loaded to all brokers metadata cache buffer.

#2: You don't have to configure any kind of proxy servers for kafka cluster. It can manage itself to implement the fail-over and load-balancing.

#3: Just assigning an unused ID and data folder is enough like the way you did for those two brokers.

Also remember, because ZAB requires majority, it is advised to use an odd number of machines as the zookeeper quorum.

Hope this will work

Join our Kafka course online today to become certified expert.

Thank You