Riemann Workshop Materials

Last year we held a Riemann workshop for our developers. The materials for the workshop were open-sourced but I never actually got around to promoting them in any way so they've just languished, like that bag of salad at the back of the fridge, slowly softening to a soggy senescence.

Since I've been writing about Rsyslog and Riemann, though, it seemed like a good time to resurrect my hard work and share it more widely. This workshop will walk you through monitoring both technical metrics and business metrics using Riemann and Collectd, plus storing and querying them with Influx, and


Custom Metrics with Rsyslog and Riemann

Back in December I said I was interested in replacing Logstash with Rsyslog, but that we needed a Riemann module to cover some of our existing functionality. Specifically we send metrics to Riemann from Logstash for three reasons:

  1. We send internal metrics from Logstash to monitor how events flow through our log pipeline.
  2. We forward all ERROR and CRITICAL logs to Riemann, which performs roll-up and throttling. Errors are forwarded to Slack, and Criticals are sent to Pagerduty.
  3. We allow developers to send application metrics in their structured log.

After some leisurely hacking over the last few days, I've got


Shipping Dynstats Metrics with Omriemann

It's been a little longer than I expected, but I'm finally back and working on Rsyslog. Last time, we looked at how impstats could be used to generate internal metrics from Rsyslog, and how to alert on those metrics in Riemann.

This time I want to look at how the Dynstats module in Riemann can be used to do more interesting monitoring of our applications.
Grafana: Error Rate Graph The Dynstats module provides a simple interface for counting events in Rsyslog. Any time we see a particular pattern in our logs, or we receive a log file from a particular application, we can increment


Monitoring with Rsyslog and Riemann

In a previous post I said I was interested in replacing Logstash with Rsyslog in our ELK stack. I've been working on one of the outstanding items from that project - the ability to send metrics to Riemann. The Made.com fork of Rsyslog now has an alpha-quality version of the Riemann output module. Although there's still some missing functionality for more advanced scenarios, we can already do interesting things with the module as it stands.

For our first example, let's imagine that we want to trigger an alert if our log volume changes drastically. For example, we may stop