Custom Metrics with Rsyslog and Riemann

Back in December I said I was interested in replacing Logstash with Rsyslog, but that we needed a Riemann module to cover some of our existing functionality. Specifically we send metrics to Riemann from Logstash for three reasons:

  1. We send internal metrics from Logstash to monitor how events flow through our log pipeline.
  2. We forward all ERROR and CRITICAL logs to Riemann, which performs roll-up and throttling. Errors are forwarded to Slack, and Criticals are sent to Pagerduty.
  3. We allow developers to send application metrics in their structured log.

After some leisurely hacking over the last few days, I've got


Shipping Dynstats Metrics with Omriemann

It's been a little longer than I expected, but I'm finally back and working on Rsyslog. Last time, we looked at how impstats could be used to generate internal metrics from Rsyslog, and how to alert on those metrics in Riemann.

This time I want to look at how the Dynstats module in Riemann can be used to do more interesting monitoring of our applications.
Grafana: Error Rate Graph The Dynstats module provides a simple interface for counting events in Rsyslog. Any time we see a particular pattern in our logs, or we receive a log file from a particular application, we can increment


Monitoring with Rsyslog and Riemann

In a previous post I said I was interested in replacing Logstash with Rsyslog in our ELK stack. I've been working on one of the outstanding items from that project - the ability to send metrics to Riemann. The fork of Rsyslog now has an alpha-quality version of the Riemann output module. Although there's still some missing functionality for more advanced scenarios, we can already do interesting things with the module as it stands.

For our first example, let's imagine that we want to trigger an alert if our log volume changes drastically. For example, we may stop


REK it.

At we're unabashed fans of the ELK stack, and I spend a decent amount of my time thinking about how we parse, ship, and store logs.

Currently, we use an ELK stack setup that looks like this:

Current setup

Rsyslog receives logs from our Docker containers, via the syslog docker logging driver, and from the rest of the system via the journal. We tag the logs at this point (as application, http, or system logs), and normalise them all to the same json format.

We ship those json logs to an Elasticache Redis instance, and consume them in Logstash. Finally,