I have created two brokers having id 0 and 1 and one consumer group having one consumer. When I start the consumer, it is electing broker 1 as group coordinator.
To validate the failover scenarios, I have stopped broker 1 and now consumer is able to elect broker 0 as group coordinator but when I start the broker 1 and stop the broker 0 consumer is not able to elect broker 1 as group coordinator and in the logs of broker 1 we are getting the below error.
reason : removing member on heartbeat expiration
And on consumer side we keep getting below error.
09:56:26.473 [main] WARN o.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-consumer_bankstmt_group-1, groupId=consumer_bankstmt_group] Connection to node 0 could not be established. Broker may not be available.
09:56:26.474 [main] INFO o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-consumer_bankstmt_group-1, groupId=consumer_bankstmt_group] Error sending fetch request (sessionId=501703391, epoch=INITIAL) to node 0: {}.
org.apache.kafka.common.errors.DisconnectException: null