Storm Monitoring using JMX-JMXTrans-Ganglia

Though Storm supports a full-fledged UI, some applications where Ganglia is being used as a kind of universal tool for displaying the metrics of the nodes in clusters of various technologies present in the application, it's essential to have Storm cluster nodes also enabled to report their metrics to Ganglia. 

Since as of yet, there is no in-built support in Storm like we have in Hadoop, HBase etc that might enable its monitoring using Ganglia, we need to do that using JMXTrans. This post is about how to setup a JMX cluster and configure Storm Cluster nodes so that the target can be achieved.

Setting up JMXTrans: 

Follow the steps below to setup a JMXTrans cluster that would act as a bridge between   the Storm Cluster nodes and Ganglia and form the reporting channel for Ganglia.

• Obtain the setup jmxtrans_20121016-175251-ab6cfd36e3-1_all.deb and extract it on all the machines of the Storm cluster

• Copy /path_to_extracted_jmx_setup/jmxtrans_20121016-175251-ab6cfd36e3-01_all/data/usr/share/jmxtrans to /usr/share/jmxtrans 

• Now any .json can be run using the following command 

/usr/share/jmxtrans/jmxtrans.sh start /path_to_json/example.json

• And jmxtrans can be stopped using 

/usr/share/jmxtrans/jmxtrans.sh stop

Configure the storm daemons to report to JMXTrans, add the following to ~/.storm/storm.yaml

“storm.yaml”- Supervisor Nodes

Add the following to the "conf/storm.yaml" on all the supervisor nodes of the cluster

worker.childopts: " -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1%ID%"

Now, json file can send the metrics from ports 16700, 16701, 16702, 16703. Also add the following to report the metrics of the jvm running the supervisor

supervisor.childopts: " -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=any_open_port_number"

“storm.yaml”- Nimbus

Specify just the following, however the presence of the above entries will not affect its performance.

nimbus.childopts: " -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=any_open_port_number"

Rest needs to be done in the json files. Storm cluster machines do not need ganglia monitoring daemons(gmond) to be running on all nodes, they can also report to a remote gmond. Finally, run the json files as mentioned above on each of the storm cluster nodes you want to monitor.

Sample JSON- Storm Workers

{   "servers" : [ {     "port" : "16700", <--- Defined Storm JMX Port      "host" : "127.0.0.1",  <--- Storm Worker     "queries" : [ {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",         "settings" : {             "groupName" : "workerMemory",           "host" : "ip_of_gmond_server",           "port" : 8649,             "v3.1" : false         }       } ],       "obj" : "java.lang:type=ClassLoading",       "attr" : [ "LoadedClassCount", "UnloadedClassCount" ]     } ],     "numQueryThreads" : 2   } ] }

Sample JSON- Storm Supervisors

{   "servers" : [ {     "port" : "assigned_port_no for eg. 10000", <---Defined Storm JMX Port   "host" : "127.0.0.1", <--- Storm Supervisor     "queries" : [ {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",         "settings" : {             "groupName" : "SuperVisorMemory",           "host" : "ip_of_gmond_server",           "port" : 8649,             "v3.1" : false         }       } ],       "obj" : "java.lang:type=Memory",       "resultAlias": "supervisor",       "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]     }],     "numQueryThreads" : 2   } ] }

Sample JSON- Storm Nimbus

{   "servers" : [ {     "port" : "assigned_port_no", <---Defined Storm JMX Port      "host" : "127.0.0.1",  <--- Storm Nimbus     "queries" : [ {       "outputWriters" : [ {         "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",         "settings" : {             "groupName" : "NimbusMemory",           "host" : "ip_of_gmond_server",           "port" : 8649,             "v3.1" : false         }       } ],       "obj" : "java.lang:type=Memory",       "resultAlias": "nimbus",       "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]     }],     "numQueryThreads" : 2   } ] }

With the help of the above JSON sample files your Storm Cluster nodes can start reporting their metrics onto the Ganglia Web UI. 

All the best !!!