Deploy Apache Kafka and Zookeeper Cluster on Kubernetes

This document will gives you how to deploy 3 nodes kafka and 3 node zookeeper cluster on kubernetes. I believe this will help you because most of the online documents are there with “helm” and I couldn’t fine simple way to deploy these two component together.

Prerequisites

1. kubernetes cluster with kubectl access.

2. Basic shell scripts knowledge

Lets deploy

What you have to do is just copy this yml file and deploy this on your kubernetes cluster.

Deploy command

kubectl apply -f zookeeper-kafka-cluster.yml

If your zookeeper containers didn't run properly you can check the logs

kubectl logs zookeepr-0 

also this scripts is the game changer in the bitnami zookeeper image therefore you can change it depend on your hostname format.

echo $(( $(cat /etc/hosts | grep zookeeper | awk '{print($3)}' | awk '{split($0,array,"-")} END{print array[3]}') + 1 )) > /bitnami/zookeeper/data/myid

If everything runs perfectly then you can check the pods

kubectl get pods 

Also you can access the kafka cluster with the pod dns.

Eg: kafka-0.kafka-cluster.default.svc.cluster.local:9092

Finally I have used binami images here and this is really easy for me to deploy the cluster the way I want.

I hope I have save your day… enjoy.. :D