Effective GlusterFs monitoring using hooks

Let us imagine we have a GlusterFs monitoring system which displays list of volumes and its state, to show the realtime status, monitoring app need to query the GlusterFs in regular interval to check volume status, new volumes etc. Assume if the polling interval is 5 seconds then monitoring app has to run gluster volume info command ~17000 times a day!

How about maintaining a state file in each node? which gets updated after every new GlusterFs event(create, delete, start, stop etc).

In this blog post I am trying to explain the possibility of creating state file and using it.

As of today GlusterFs provides following hooks, which we can use to update our state file.

    create
    delete
    start
    stop
    add-brick
    remove-brick
    set

How to use hooks

GlusterFs hooks present in /var/lib/glusterd/hooks/1 directory. Following example shows sending message to all users using wall command when any new GlusterFs volume is created.

Create a shell script /var/lib/glusterd/hooks/1/create/post/SNotify.bash and make it executable. Whenever a volume is created GlusterFs executes all the executable scripts present in respective hook directory(Glusterfs executes only the scripts which filename starting with 'S')

#!/bin/bash
VOL=
ARGS=$(getopt -l "volname:"  -name "" $@)
eval set -- "$ARGS"
while true; do
    case $1 in
        --volname)
            shift
            VOL=$1
            ;;
        *)
            shift
            break
            ;;
    esac
    shift
done

wall "Gluster Volume Created: $VOL";

Experimental project - GlusterWeb

This experimental project maintains a sqlite database /var/lib/glusterd/nodestate/glusternodestate.db which gets updated after any GlusterFs event. For example if a GlusterFs volume is created then it updates volumes table and also bricks table.

This project depends on glusterfs-tools so install both projects.

git clone https://github.com/aravindavk/glusterfs-tools.git
cd glusterfs-tools
sudo python setup.py install

git clone https://github.com/aravindavk/glusterfs-web.git
cd glusterfs-web
sudo python setup.py install

By running setup, this tool will install all the hooks which are required for monitoring. (cleanup is for removing all the hooks)

sudo glusternodestate setup

All set! now run glusterweb to start webapp.

sudo glusterweb

Web application starts running in http://localhost:8080 you can change the port using --port or -p option.

sudo glusterweb -p 9000

Initial version of web interface.

Future plans

Authentication: Option to provide username and password or access key while running glusterweb, For example

sudo glusterweb --username aravindavk --password somesecret
# or
sudo glusterweb --key secretonlyiknow

More gluster hooks support: we need more GlusterFs hooks for better monitoring(refer Problems below)

More GlusterFs features support: As a experiment UI only lists volumes, we need improved UI and support for different gluster features.

Actions support: Support for volume creation, adding/removing bricks etc.

REST api and SDK: Providing REST api for gluster operations.

Many more: Not yet planned :)

Problems

State file consistency: If glusterd goes down in the node then the database will have wrong details about node’s state. One workaround is to reset the database if glusterd is down using a cron job, when glusterd comes up, database will not gets updated and the database will have previous updated details. To prevent this we need a glusterfs hook for glusterd-start.

More hooks: As of today we don’t have hooks for volume down/up, brick down/up and other events. We need following hooks for effective monitoring glusterfs.(Add more if anything missing in the list)

    glusterd-start
    peer probe
    peer detach
    volume-down
    volume-up
    brick-up
    brick-down

Let me know your thoughts! Thanks.