Kafka itself comes with command line tools that can do all the administration tasks, but those tools aren’t very convenient because they are not integrated into one tool and you need to run a different tool for different tasks. Moreover, it is getting difficult to work with them when your clusters grow large or when you have several clusters.
So today we will cover some GUI alternatives.
First is Kafka tool. It is a windows program that can connect to a Kafka cluster and do all basic tasks. It can list brokers, topics or consumers and their properties. It allows you to create new topics or update existing ones and you can even look at the messages in a topic or partition. Although it is very useful, its UI seems somewhat old and it lacks some monitoring features like topic lag.
Kafka manager is a web based powerful management system for Kafka developed at Yahoo!. It is capable of administrating multiple clusters, it can show statistics on individual brokers or topics such as messages per second, lag ans so on. I think this is the best tool out there at the moment.
Here are the steps to install it:
First, download the zip distribution from the link above and unzip it.
Edit application.conf and change kafka-manager.zkhosts to one or more of your zookeeper hosts, for example kafka-manager.zkhosts=”cloudera2:2181″.
Now you should build kafka manager. It uses play framework but it is installed and configured automatically (unlike Kafka web console that is discussed later). In the directory where you unzipped it, run:
./sbt clean dist
This can take long time to complete (about 30 minutes).
This will create a distribution file ./target/universal/kafka-manager-188.8.131.52.zip
Unzip it to a directory of your choice.
Before we run it, there seems to have a bug that prevents kafka-manager to automatically pick up the configuration file when starting (as reported here). The workaround is to explicitly provide the configuration file at startup:
export KAFKA_MANAGER_HOME=/home/kafka/kafka-manager-184.108.40.206 export PATH=$PATH:$KAFKA_MANAGER_HOME/bin Kafka-manager –Dconfig.file=$KAFKA_MANAGER_HOME/conf/application.conf
After it’s done starting, you can access it with you browser at port 9000 (this is the default, but you can change it by adding Dhttp.port= startup parameter).
Few words about Kafka web console that seems to be the predecessor of Kafka manager: Like Kafka manager, it is built on top of Play framework, but its installation requires you to install Play manually. The UI seems very nice but the project page says it is no longer maintained and supported and advise to consider using Kafka manager instead.
Kafdrop seems very impressive and its capabilities are very similar to those of Kafka manager. It worked just fine when I tested it but this is an early stage product that you should thoroughly examine before using in production.
After downloading it you will have to build it:
First, install maven (if you don’t have it already):
wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo yum install apache-maven
Then build kafdrop:
unzip Kafdrop-master.zip cd Kafdrop-master mvn clean package
This may take some time to complete. When its done, you will find a directory named “target” and inside it a file named “kafdrop-1.1.3-SNAPSHOT.jar”.
To run it, just type:
java -jar kafdrop-1.1.3-SNAPSHOT.jar --zookeeper.connect=<host:port>, <host:port>...
The default listening port is 9000, but you can change it by adding –server.port=<port> parameter.
Another tool worth mentioning is burrow from Linkedin. We will not cover it in detail this time because it does not fall into the same category as the previous tools mentioned here. It does not have a graphical user interface and it does not have any cluster management capabilities. It’s strength is in its ability to monitor consumer groups for lags and general health in a simple way. It can also automatically notify an administrator by email or http about any problems. I covered it in more detail in this post: Monitoring Kafka consumer lag with Burrow.