Using SNMP, I have been creating NFS graphs like below for all of our NetApp Filers using RRDTool

Ever since I started running NFS on Linux servers in addition to the Netapps, I wanted to graph the Linux nfs traffic. The RedHat distro comes with net-snmp package. While the package does not have any specific MIB support for NFS, it is infinitely extensible using custom external programs. Practically anything can be pushed/pulled via SNMP. So I decided to write a perl script (well, it started out as a shell script) to convert nfsstats into SNMP values.

The result is proc2snmp.pl. In addition to reporting the nfsd statistics reported by the system, the script also reports another statistical value “v3total”, which is the total number of all NFS calls (since last time the nfsd process has started). I am using this value to calculate percentages of each type of call for graphing purposes.

Download

proc2snmp.pl - proc2snmp.pl utility
gathernfs.sh  - utility to poll snmp and populate the RRD database

SNMP Configuration

Setting up snmp can be as easy as running /usr/bin/snmpconf utility, which reads the existing files and generates a new snmpd.conf after asking a set of questions. One can also manually edit the /etc/snmp/snmpd.conf file and put in the directives. A basic snmpd.conf file is below:

[root@nfsa ~]# grep -v “#” /etc/snmp/snmpd.conf  | grep -v ‘^$’
pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat
pass .1.3.6.1.4.1.2021.255 /usr/local/custom/proc2snmp.pl
syslocation  “2nd belt, 4th asteroid - beetlegeuse”
syscontact  noreply (at) blogsome (dot) com
sysservices 72
proc  nfsd 32 8
rocommunity  public

The read-only community string is “public” and we are using net-snmp pass directive to pass the control of the entire MIB tree of .1.3.6.1.4.1.2021.255 to an external program called /usr/local/custom/proc2snmp.pl. The good thing about the pass directives is that the ENTIRE subtree specified in the line is available for use by the program, which allows for future expansion - which we know always happens.

The following line in the snmpd.conf file is needed:

pass .1.3.6.1.4.1.2021.255 /usr/local/custom/proc2snmp.pl

Install the perl script in the specified location and you are ready to test!

Testing

#snmpwalk -v2c -c public localhost 1.3.6.1.4.1.2021.255
UCD-SNMP-MIB::ucdavis.255.8 = STRING: “so long and thanks for all the fish”
UCD-SNMP-MIB::ucdavis.255.8.1 = STRING: “/usr/local/custom/proc2snmp.pl”
UCD-SNMP-MIB::ucdavis.255.8.2.1 = STRING: “v3null”
UCD-SNMP-MIB::ucdavis.255.8.2.2 = Gauge32: 974
UCD-SNMP-MIB::ucdavis.255.8.3.1 = STRING: “v3getattr”
UCD-SNMP-MIB::ucdavis.255.8.3.2 = Gauge32: 17139386
UCD-SNMP-MIB::ucdavis.255.8.4.1 = STRING: “v3setattr”
UCD-SNMP-MIB::ucdavis.255.8.4.2 = Gauge32: 6848261
UCD-SNMP-MIB::ucdavis.255.8.5.1 = STRING: “v3lookup”
UCD-SNMP-MIB::ucdavis.255.8.5.2 = Gauge32: 6543679
UCD-SNMP-MIB::ucdavis.255.8.6.1 = STRING: “v3access”
UCD-SNMP-MIB::ucdavis.255.8.6.2 = Gauge32: 2918027
UCD-SNMP-MIB::ucdavis.255.8.7.1 = STRING: “v3readlink”
UCD-SNMP-MIB::ucdavis.255.8.7.2 = Gauge32: 361
UCD-SNMP-MIB::ucdavis.255.8.8.1 = STRING: “v3read”
UCD-SNMP-MIB::ucdavis.255.8.8.2 = Gauge32: 1381807
UCD-SNMP-MIB::ucdavis.255.8.9.1 = STRING: “v3create”
UCD-SNMP-MIB::ucdavis.255.8.9.2 = Gauge32: 845070
UCD-SNMP-MIB::ucdavis.255.8.10.1 = STRING: “v3mkdir”
UCD-SNMP-MIB::ucdavis.255.8.10.2 = Gauge32: 22960
UCD-SNMP-MIB::ucdavis.255.8.11.1 = STRING: “v3write”
UCD-SNMP-MIB::ucdavis.255.8.11.2 = Gauge32: 6218351
UCD-SNMP-MIB::ucdavis.255.8.12.1 = STRING: “v3symlink”
UCD-SNMP-MIB::ucdavis.255.8.12.2 = Gauge32: 185
UCD-SNMP-MIB::ucdavis.255.8.13.1 = STRING: “v3mknod”
UCD-SNMP-MIB::ucdavis.255.8.13.2 = Gauge32: 0
UCD-SNMP-MIB::ucdavis.255.8.14.1 = STRING: “v3remove”
UCD-SNMP-MIB::ucdavis.255.8.14.2 = Gauge32: 113995
UCD-SNMP-MIB::ucdavis.255.8.15.1 = STRING: “v3rmdir”
UCD-SNMP-MIB::ucdavis.255.8.15.2 = Gauge32: 2815
UCD-SNMP-MIB::ucdavis.255.8.16.1 = STRING: “v3rename”
UCD-SNMP-MIB::ucdavis.255.8.16.2 = Gauge32: 10830
UCD-SNMP-MIB::ucdavis.255.8.17.1 = STRING: “v3link”
UCD-SNMP-MIB::ucdavis.255.8.17.2 = Gauge32: 26781
UCD-SNMP-MIB::ucdavis.255.8.18.1 = STRING: “v3readdir”
UCD-SNMP-MIB::ucdavis.255.8.18.2 = Gauge32: 73914
UCD-SNMP-MIB::ucdavis.255.8.19.1 = STRING: “v3readdirplus”
UCD-SNMP-MIB::ucdavis.255.8.19.2 = Gauge32: 139878
UCD-SNMP-MIB::ucdavis.255.8.20.1 = STRING: “v3fsstat”
UCD-SNMP-MIB::ucdavis.255.8.20.2 = Gauge32: 479
UCD-SNMP-MIB::ucdavis.255.8.21.1 = STRING: “v3fsinfo”
UCD-SNMP-MIB::ucdavis.255.8.21.2 = Gauge32: 827
UCD-SNMP-MIB::ucdavis.255.8.22.1 = STRING: “v3pathconf”
UCD-SNMP-MIB::ucdavis.255.8.22.2 = Gauge32: 1
UCD-SNMP-MIB::ucdavis.255.8.23.1 = STRING: “v3commit”
UCD-SNMP-MIB::ucdavis.255.8.23.2 = Gauge32: 4491773
UCD-SNMP-MIB::ucdavis.255.8.24.1 = STRING: “v3total”
UCD-SNMP-MIB::ucdavis.255.8.24.2 = Gauge32: 46780946

Notice how I am using a subtree 1.3.6.1.4.1.2021.255.8 for nfs v3 server statistics. This way, the script can be expanded to include other status under a different OID in the same MIBOID without additional snmp reconfigurations and restarts.

Once this is setup, the next task is to collect poll the host for statistics and stick them in a database. RRDTool is a mostly commonly used tool for these purposes. RRDTool also makes generating graphs from its databases a breeze.

Setting up a RRD Database

RRDTool installation is very well covered on the RRDTool website, so there is no need to go into it again. Ben Rockwood has an excellent article titled “Getting started with RRDTool” on his website. It is a must read for anyone interested in RRDTool. Assuming you have RRD already installed , the next step is to create the database to collect all of the nfsv3 variables.

#/usr/local/bin/rrdtool create nfsd.rrd \
        DS:v3null:GAUGE:600:0:100 \
        DS:v3getattr:GAUGE:600:0:100 \
        DS:v3setattr:GAUGE:600:0:100 \
        DS:v3lookup:GAUGE:600:0:100 \
        DS:v3access:GAUGE:600:0:100 \
        DS:v3readlink:GAUGE:600:0:100 \
        DS:v3read:GAUGE:600:0:100 \
        DS:v3write:GAUGE:600:0:100 \
        DS:v3create:GAUGE:600:0:100 \
        DS:v3mkdir:GAUGE:600:0:100 \
        DS:v3symlink:GAUGE:600:0:100 \
        DS:v3mknod:GAUGE:600:0:100 \
        DS:v3remove:GAUGE:600:0:100 \
        DS:v3rmdir:GAUGE:600:0:100 \
        DS:v3rename:GAUGE:600:0:100 \
        DS:v3link:GAUGE:600:0:100 \
        DS:v3readdir:GAUGE:600:0:100 \
        DS:v3readdirplus:GAUGE:600:0:100 \
        DS:v3fsstat:GAUGE:600:0:100 \
        DS:v3fsinfo:GAUGE:600:0:100 \
        DS:v3pathconf:GAUGE:600:0:100 \
        DS:v3commit:GAUGE:600:0:100 \
        DS:v3total:GAUGE:600:0:100 \
        RRA:AVERAGE:0.5:1:600 \
        RRA:AVERAGE:0.5:24:775 \
        RRA:AVERAGE:0.5:288:797 \
        RRA:MAX:0.5:1:600 \
        RRA:MAX:0.5:6:600\
        RRA:MAX:0.5:24:775\
        RRA:MAX:0.5:288:797

This will generate a file called nfsd.rrd, which is an RRD database in the current working directory.

Populating RRD Database

Populating the rrd database is done using the rrdtool update command. I am interested in the percentages of the nfs calls so I wrote a shell script to poll SNMP for the absolute values and calculate the percentages. The script can also be downloaded from the Download section. The script is self-explanatory and it is kinda documented. It can be installed as a cron job to run every 5 mins and update the database.

Generating graphs from the RRD Database

To verify the RRD database is getting populated, rrdtool info can be run to view the contents of the database.

#rrdtool info nfsd.rrd |more
filename = “nfsd.rrd”
rrd_version = “0003″
step = 300
last_update = 1153770616
ds[v3null].type = “GAUGE”
ds[v3null].minimal_heartbeat = 600
ds[v3null].min = 0.0000000000e+00
ds[v3null].max = 1.0000000000e+02
ds[v3null].last_ds = “UNKN”
ds[v3null].value = 0.0000000000e+00
ds[v3null].unknown_sec = 0
ds[v3getattr].type = “GAUGE”
ds[v3getattr].minimal_heartbeat = 600
ds[v3getattr].min = 0.0000000000e+00
ds[v3getattr].max = 1.0000000000e+02
ds[v3getattr].last_ds = “UNKN”
ds[v3getattr].value = 5.7567703662e+02
ds[v3getattr].unknown_sec = 0
~output truncated for brevity

Good - Let’s generate a graph displaying the percentage of getattr calls.

# rrdtool graph /usr/local/web/rrd/images/getattr.png \
 –start=end-86400 \
 –title=’NFS getattr’ \
 –vertical-label=’% of all NFS calls’ \
 –imgformat=PNG \
 –width=500 \
 –base=1000 \
 –height=120 \
 –alt-y-mrtg \
 –alt-autoscale \
 –interlaced \
 –slope-mode \
 -c BACK#000000 -c CANVAS#000000 -c GRID#bbbbbb -c MGRID#CCCCCC -c FONT#CCCCCC \
 -c FRAME#000000 -c AXIS#FFFFFF -c SHADEA#000000 -c SHADEB#000000 -c ARROW#C9B215 \
 DEF:b1=/usr/local/web/rrd/data/ibrix1.rrd:v3getattr:AVERAGE \
 DEF:b2=/usr/local/web/rrd/data/ibrix1.rrd:v3getattr:MAX \
 AREA:b1#33FFFF77:v3getattr  \
 VDEF:b1_AVERAGE=b1,AVERAGE \
 GPRINT:b1_AVERAGE:”Avg\: %8.2lf%s” \
 VDEF:b1_MAX=b1,MAXIMUM \
 GPRINT:b1_MAX:”Max\: %8.2lf%s\n” \
 LINE2:b2#00FF00

If everything went well, it is generate a rather anti-climactical output such as:

597x213

which is nothing but the dimesions of the image it just created. And the image looks like this:

The power of RRDTool lies not in graphing 1 measly data point such as getattr, but in graphing multiple datapoints as the first graph shows. Also, there are many front end tools that offer web-based graph generation and display. One such tool that I use extensively is drraw. Drraw allows you to generate graphing templates, dashboards to display multiple graphs on a single webpage etc.

Links

It just so happens that there are two people on the Internet, thinking the same thing and writing the same piece of code at the same time! Macdee just posted on sourceforge, a version of nfsstats.pl that does the same thing, slightly differently.

Technorati Tags: linux, snmp, nfs, ibrix, net-snmp, proc2snmp, nfsstat