Kafka Monitoring using JMX-JMXTrans-Ganglia

Monitoring Kafka Clusters using Ganglia is a matter of a few steps. This blog post lists down those steps with an assumption that you have your Kafka Cluster ready.

Step-I: Setup JMXTrans on all the machines of the Kafka cluster as done on the Storm cluster in the previous post.

Step-II: In the kafka setup, edit “kafka-run-class.sh” script file by adding the following line to it:

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false "

Step-III: Also, edit the “kafka-server-start.sh” script file present in the kafka setup to set the JMX port to 9999 by adding the following line:

export JMX_PORT=${JMX_PORT:-9999}

Now, on all the nodes of the cluster on which you have performed the above steps, you can run the following json file after which it should start reporting its metrics to the Ganglia server.

Sample JSON file

Run the code below in the form of a json file using the following command:

/usr/share/jmxtrans/jmxtrans.sh start /path_to_sample_json/example.json

Note: Please change the paths of output files in the code below to paths accessible on your cluster machines.

{   "servers" : [ {     "port" : "9999", <--- Defined Kafka JMX Port     "host" : "127.0.0.1",  <--- Kafka Server     "queries" : [ {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",           "settings" : {                    "outputFile" : "/home/jayati/JMXTrans/kafkaStats/bufferPool_direct_stats.txt",                    "v31" : false                    }       } ],       "obj" : "java.nio:type=BufferPool,name=direct",       "resultAlias": "bufferPool.direct",       "attr" : [ "Count", "MemoryUsed", "Name", "ObjectName", "TotalCapacity" ]     }, {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",         "settings" : {                    "outputFile" : "/home/jayati/JMXTrans/kafkaStats/bufferPool_mapped_stats.txt",                    "v31" : false                    }       } ],       "obj" : "java.nio:type=BufferPool,name=mapped",       "resultAlias": "bufferPool.mapped",       "attr" : [ "Count", "MemoryUsed", "Name", "ObjectName", "TotalCapacity" ]     }, {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",         "settings" : {                    "outputFile" : "/home/jayati/JMXTrans/kafkaStats/kafka_log4j_stats.txt",                    "v31" : false                    }       } ],       "obj" : "kafka:type=kafka.Log4jController",       "resultAlias": "kafka.log4jController",       "attr" : [ "Loggers" ]     }, {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",           "settings" : {                    "outputFile" : "/home/jayati/JMXTrans/kafkaStats/kafka_socketServer_stats.txt",                    "v31" : false                    }       } ],       "obj" : "kafka:type=kafka.SocketServerStats",       "resultAlias": "kafka.socketServerStats",       "attr" : [ "AvgFetchRequestMs", "AvgProduceRequestMs", "BytesReadPerSecond", "BytesWrittenPerSecond", "FetchRequestsPerSecond", "MaxFetchRequestMs", "MaxProduceRequestMs" , "NumFetchRequests" , "NumProduceRequests" , "ProduceRequestsPerSecond", "TotalBytesRead", "TotalBytesWritten", "TotalFetchRequestMs", "TotalProduceRequestMs" ]     } ],     "numQueryThreads" : 2   } ] }

}
Get high on the Ganglia graphs showing your Kafka Cluster metrics. :) 
All the best !!!