Log in

No account? Create an account
entries friends calendar profile Previous Previous Next Next
RRDtool is amazing - Ed's journal
RRDtool is amazing
I've been playing recently with a piece of software that I keep coming back to, and discovering new coolness.
It's called RRDtool.

RRD stands for Round Robin Database. And what this tool does is allow you to insert time based statistics into an RRD, and extract them later as graphs. It includes automatic statistic aggregating and archiving, which means it's ideal for ... well, all sorts of statistics really.

It's used by a spectacular number of utilities - including Cacti, MRTG, and - the way I first ran into it - Big Brother. Here's the full list.

But the tool itself it really very useful - there's all sorts of things that have performance counters, and ... RRDtool is almost perfectly suited to collating them - allowing you to sample information at almost any frequency you choose - and then collate them from high resolution diagnostics, to longer term trends.

It's easier to get going than you might think - first of all, you create an RRD. You do so by giving it a 'Data Series' (DS) - which defines the input data. And an Round Robin Archive (RRA) - which defines resolution, retention and auto-archiving.

And then you insert into your RRD samples collected, which it turns into consolidated data points - which can be extracted and turned into graphs (very easily, but you've got a very powerful graphing tool that also allows you to add in formula and transformation if you so desire). Or just pulled out as a set of data points.

But the reason I've been particularly impressed with it recently, is because it implements something called 'Holt Winters forecasting'. Now, to save too much brain ache, what this is is a technique for smoothing off a graph. But the important part is that it includes a seasonal variance (and by 'seasonal' it means in a statistical sense - in most cases in IT, as 'season' is a day) which means you've a mechanism to smooth and predict - along with an expected variance, based on your seasonal trend.

This means that - rather than setting a threshold of 'bad' and 'good' on your system state (which rarely works well, because a lot of system statistics are very hard to give such a binary answer) - you can instead detect aberrant behavior - simply count (on a rolling window) the number of times your measurement strays outside the expected variance, and flag an error if it does.
This is really very cool indeed. Self tuning statistics that can bring themselves to your attention when they're 'interesting'.

I would also note - this has lead me to discover R - a statistic modeling and manipulation tool. What this has helped with, is tuning Holt Winters parameters - it has parameters for adjustment rates of statistics - as with any smoothing algorithm, you set a 'weight' for adjusting the curve. The answer to 'what should I set these parameters to' is 'it depends' normally.
What R will do is (easily) let you feed in your samples, run 'HoltWinters' on it, and spit out the optimal parameters based on your data. (It does least-squared regression to find which parameters provide a 'best fit').
This too is awesome. (And R can do amazing amounts of other stuff too, which I like).

Anyway, if - like me - you like being able to see traffic graphs, CPU loads, average/peak concurrent user trends, response times and all sorts of stats in graphical and historical form - this is exactly the tool for the job.
Leave a comment