JMX timeseries data
When looking at monitoring Java applications it seems that JMX is still one of the best standard ways to get information from your application and application servers.
All containers, and most frameworks publish MBeans into the JMX context, but the tools for processing and retrieving them still seem to be limited.
The quickest and easiest solution is to fire up VisualVM, install the MBeans plugin and connect to your container. You’ll end up with access to a heap of values, most of which won’t help you with your problem, but there may be some pieces of gold in there that are key to helping you identify issues or performance problems.
Not matter what container you run, you’ll get access to the key Memory and Threading beans. These are probably a great place to start when looking at a system in abnormal state. But how do you know what is normal and what is not. The best way is to regularly check on some key metrics and over time you will learn what is normal and what is abnormal.
So how do you do this? You could just fire up VisualVM every now and then and read the values, hopefully you’ll remember what looks normal or not. Or you can setup something that extracts the JMX values periodically and stores them.
If you’ve been following me on Twitter then you’ll know I have been recently using Librato, and that it is a really simple and directed tool that produces so much power by how flexible it is.
Librato is a simple website that enables you to push in timestamped JSON objects, and then graph them on a timeseries graph. This may sound simple, but one of the most complex parts of Metrics analysis is managing the volume of information and drawing relationships between metrics. The Librato platform allows you to easily build ‘Instruments’ that combine multiple different metrics against the same time frame. This simple tool enables a multitude of analysis to be done.
So we now have the raw data in the JMX beans, and a place to store it in Librato, but the missing piece is how to get the data from one into the other.
Of course the Librato guys have build a connector to extract specific JMX metrics and import them into the Librato platform. The code is available on GitHub and pretty self explantory.
Just configure your Librato email and token, and supply the connection credentials for the JMX datasource and you are almost there.
1 2 3 4 5 6 7 8 9
librato-metrics-tap-jmxbeans\ --publish \ --email "$EMAIL" \ --token "$TOKEN" \ --source "$SOURCE" \ --jmx-host "$JMX_HOST" \ --jmx-port $JMX_PORT \ --data-file-full tomcat-jmx.yaml \ -i 30
The only decision left is to identify what to extract, as a simple example here is the configuration to pull out the key Thread and Memory statistics as well as the Tomcat specific beans for extracting the Active Sessions from a ‘ROOT’ context.
1 2 3 4 5 6 7 8
--- java.lang:type=Threading: ThreadCount: PeakThreadCount: java.lang:type=Memory: HeapMemoryUsage: Catalina:type=Manager,context=/,host=localhost: activeSessions:
Note: At the time of writing this the Librato library to process that yaml file needs to be patched, the ‘context=/’ causes issues trying to post them to the Librato API, but worry not, there is a patch available that fixes your problems.. Hopefully soon the Pull Request will get patched and it will be even easier to use.
I also ran into an issue with the library, in that it did not support JMX Authentication, but that was an easy fix and there is now another Pull Request to add support for a –jmx-username and –jmx-password attribute to enable connection to authenticated JMX endpoints.
Monitoring now made easy, enabling you to quickly extract key JMX metrics, pump them into Librato and perform analysis.