Kafka Monitoring using JMX-JMXTrans-Ganglia

Monitoring Kafka Clusters using Ganglia is a matter of a few steps. This blog post lists down those steps with an assumption that you have your Kafka Cluster ready.

Step-I: Setup JMXTrans on all the machines of the Kafka cluster as done on the Storm cluster in the previous post.

Step-II: In the kafka setup, edit “kafka-run-class.sh” script file by adding the following line to it:

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false "

Step-III: Also, edit the “kafka-server-start.sh” script file present in the kafka setup to set the JMX port to 9999 by adding the following line:

export JMX_PORT=${JMX_PORT:-9999}

Now, on all the nodes of the cluster on which you have performed the above steps, you can run the following json file after which it should start reporting its metrics to the Ganglia server.

Sample JSON file

Run the code below in the form of a json file using the following command:

/usr/share/jmxtrans/jmxtrans.sh start /path_to_sample_json/example.json

Note: Please change the paths of output files in the code below to paths accessible on your cluster machines.

{   "servers" : [ {     "port" : "9999", <--- Defined Kafka JMX Port     "host" : "127.0.0.1",  <--- Kafka Server     "queries" : [ {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",           "settings" : {                    "outputFile" : "/home/jayati/JMXTrans/kafkaStats/bufferPool_direct_stats.txt",                    "v31" : false                    }       } ],       "obj" : "java.nio:type=BufferPool,name=direct",       "resultAlias": "bufferPool.direct",       "attr" : [ "Count", "MemoryUsed", "Name", "ObjectName", "TotalCapacity" ]     }, {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",         "settings" : {                    "outputFile" : "/home/jayati/JMXTrans/kafkaStats/bufferPool_mapped_stats.txt",                    "v31" : false                    }       } ],       "obj" : "java.nio:type=BufferPool,name=mapped",       "resultAlias": "bufferPool.mapped",       "attr" : [ "Count", "MemoryUsed", "Name", "ObjectName", "TotalCapacity" ]     }, {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",         "settings" : {                    "outputFile" : "/home/jayati/JMXTrans/kafkaStats/kafka_log4j_stats.txt",                    "v31" : false                    }       } ],       "obj" : "kafka:type=kafka.Log4jController",       "resultAlias": "kafka.log4jController",       "attr" : [ "Loggers" ]     }, {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",           "settings" : {                    "outputFile" : "/home/jayati/JMXTrans/kafkaStats/kafka_socketServer_stats.txt",                    "v31" : false                    }       } ],       "obj" : "kafka:type=kafka.SocketServerStats",       "resultAlias": "kafka.socketServerStats",       "attr" : [ "AvgFetchRequestMs", "AvgProduceRequestMs", "BytesReadPerSecond", "BytesWrittenPerSecond", "FetchRequestsPerSecond", "MaxFetchRequestMs", "MaxProduceRequestMs" , "NumFetchRequests" , "NumProduceRequests" , "ProduceRequestsPerSecond", "TotalBytesRead", "TotalBytesWritten", "TotalFetchRequestMs", "TotalProduceRequestMs" ]     } ],     "numQueryThreads" : 2   } ] }

}
Get high on the Ganglia graphs showing your Kafka Cluster metrics. :) 
All the best !!!

Storm Monitoring using JMX-JMXTrans-Ganglia

Though Storm supports a full-fledged UI, some applications where Ganglia is being used as a kind of universal tool for displaying the metrics of the nodes in clusters of various technologies present in the application, it's essential to have Storm cluster nodes also enabled to report their metrics to Ganglia. 

Since as of yet, there is no in-built support in Storm like we have in Hadoop, HBase etc that might enable its monitoring using Ganglia, we need to do that using JMXTrans. This post is about how to setup a JMX cluster and configure Storm Cluster nodes so that the target can be achieved.

Setting up JMXTrans: 

Follow the steps below to setup a JMXTrans cluster that would act as a bridge between   the Storm Cluster nodes and Ganglia and form the reporting channel for Ganglia.

• Obtain the setup jmxtrans_20121016-175251-ab6cfd36e3-1_all.deb and extract it on all the machines of the Storm cluster

• Copy /path_to_extracted_jmx_setup/jmxtrans_20121016-175251-ab6cfd36e3-01_all/data/usr/share/jmxtrans to /usr/share/jmxtrans 

• Now any .json can be run using the following command 

/usr/share/jmxtrans/jmxtrans.sh start /path_to_json/example.json

• And jmxtrans can be stopped using 

/usr/share/jmxtrans/jmxtrans.sh stop

Configure the storm daemons to report to JMXTrans, add the following to ~/.storm/storm.yaml

“storm.yaml”- Supervisor Nodes

Add the following to the "conf/storm.yaml" on all the supervisor nodes of the cluster

worker.childopts: " -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1%ID%"

Now, json file can send the metrics from ports 16700, 16701, 16702, 16703. Also add the following to report the metrics of the jvm running the supervisor

supervisor.childopts: " -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=any_open_port_number"

“storm.yaml”- Nimbus

Specify just the following, however the presence of the above entries will not affect its performance.

nimbus.childopts: " -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=any_open_port_number"

Rest needs to be done in the json files. Storm cluster machines do not need ganglia monitoring daemons(gmond) to be running on all nodes, they can also report to a remote gmond. Finally, run the json files as mentioned above on each of the storm cluster nodes you want to monitor.

Sample JSON- Storm Workers

{   "servers" : [ {     "port" : "16700", <--- Defined Storm JMX Port      "host" : "127.0.0.1",  <--- Storm Worker     "queries" : [ {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",         "settings" : {             "groupName" : "workerMemory",           "host" : "ip_of_gmond_server",           "port" : 8649,             "v3.1" : false         }       } ],       "obj" : "java.lang:type=ClassLoading",       "attr" : [ "LoadedClassCount", "UnloadedClassCount" ]     } ],     "numQueryThreads" : 2   } ] }

Sample JSON- Storm Supervisors

{   "servers" : [ {     "port" : "assigned_port_no for eg. 10000", <---Defined Storm JMX Port   "host" : "127.0.0.1", <--- Storm Supervisor     "queries" : [ {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",         "settings" : {             "groupName" : "SuperVisorMemory",           "host" : "ip_of_gmond_server",           "port" : 8649,             "v3.1" : false         }       } ],       "obj" : "java.lang:type=Memory",       "resultAlias": "supervisor",       "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]     }],     "numQueryThreads" : 2   } ] }

Sample JSON- Storm Nimbus

{   "servers" : [ {     "port" : "assigned_port_no", <---Defined Storm JMX Port      "host" : "127.0.0.1",  <--- Storm Nimbus     "queries" : [ {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",         "settings" : {             "groupName" : "NimbusMemory",           "host" : "ip_of_gmond_server",           "port" : 8649,             "v3.1" : false         }       } ],       "obj" : "java.lang:type=Memory",       "resultAlias": "nimbus",       "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]     }],     "numQueryThreads" : 2   } ] }

With the help of the above JSON sample files your Storm Cluster nodes can start reporting their metrics onto the Ganglia Web UI. 

All the best !!!