Monitoring GlusterFS - Yet another try

Monitoring is one of my favorite topics, and previously I tried multiple ways to monitor GlusterFS effectively. Initially, I decided to avoid running Gluster CLI commands with too frequent intervals by using Gluster hooks(Blog about using Gluster hooks is here). But this approach was not practical since hooks are not available for all events.

Then I started a project called "gdash" to visualize the health of the cluster using gluster command-line tools. But this was having only limited access to the cluster status and depended solely on what gluster CLI provides.

Events APIs are introduced in Gluster to fill the gaps of limited hooks support. The events listener needs to be started in each node of the cluster, and it exports the local events via registered webhooks.

Events APIs are great since it helps to reduce polling Gluster cluster. But Events APIs are not perfect because of the asynchronous nature it can miss events. If a Webhook listener is down, then there is no way to get the history of events.

Events APIs are useful for creating health/status boards but not so helpful while designing monitoring systems that require historical data to make decisions and plot graphs. For example, adding an event for each file create/delete is not practical, so events APIs are not useful for getting Volume utilization details and similar metrics.

Many approaches we tried didn’t succeed because all are dependent on gluster command-line tools. When we depend on Gluster CLIs, then collecting metrics from every node doesn’t make sense because Gluster CLIs returns the same data from all the nodes. So if we start collecting from a single node of the cluster, then we need a mechanism to choose another node when the previously active node goes down.

When we started working on the Prometheus integration, our approach was to collect metrics at the lowest level possible and use the monitoring system’s capability to aggregate the metrics from multiple nodes. For example, instead of collecting Volume utilization via mount or Gluster volume status command, Prometheus exporter collected utilization metrics from the bricks. This approach avoided all the complexities related to leader election or duplicate metrics.

The gluster-prometheus project solved problems related to some of the metrics, but not for all. We are still dependent on Gluster CLI commands for some of the metrics like Profile and others.