Custom Discovery: Discover All the Things!

Having out-grown the included Zabbix Web Interface templates, I find myself more often than not, having to create new templates as well as entirely new ways of monitoring hardware and software. I often create custom discovery rules to dynamically create new items as services and devices change. This very powerful feature of Zabbix is called Low-level Discovery or LLD for short. The feature is described as follows by Zabbix:

Low-level discovery provides a way to automatically create items, triggers, and graphs for different entities on a computer. For instance, Zabbix can automatically start monitoring file systems or network interfaces on your machine, without the need to create items for each file system or network interface manually. Additionally it is possible to configure Zabbix to remove unneeded entities automatically based on actual results of periodically performed discovery.

With LLD, we are able to automatically, at timed intervals, discover new partitions, entire SNMP OID trees, system users, network interfaces and so on. The Zabbix Agent even has some built in items keys intended to be used with LLD. Item and Trigger prototypes then handles, based on the discovered data, creation of items and triggers.

For item discovery, Zabbix Agent and SNMP have their limits. Though with the agent and a little scripting, we can create our own Custom Discovery item keys!

For the purposes of this guide, we will try to discover all files within a directory and then via prototypes, create items that read information from these files.

Note: The ‘custom.discovery’ item can be edited to move through any array and doesn’t have to ‘ls’ a directory. One could use ‘Get-Service’ on Windows to discover services or ‘ls /sys/class/net’ on Linux to discover network interfaces.

The ‘vfs.file.regexp’ item would then of course have to be edited accordingly, to retrieve the information you’re interested in for each item.

Requirements

• Zabbix 2.x and above
• Zabbix Agent on the monitored host
• Knowledge of Bash and PowerShell is a plus

How it works

Low-level Discovery is setup and the item key ‘custom.discovery’ will produce a JSON formatted list of files inside a directory of your choice with the tag ‘{#ITEM}’.

Using Item Prototypes, each ‘{#ITEM}’ object is created as items on the host(s)  with the built-in agent item key ‘vfs.file.regexp’.

In this example, ‘custom.discovery’ returns a list of absolute paths to files inside the directory ‘C:\SOME\PATH’ or ‘/SOME/PATH’. The path is hard-coded inside the Zabbix Agent configuration.

The key ‘vfs.file.regexp’ retrieves data from a file. Detailed information about the item key is available here.

vfs.file.regexp[file,regexp,<encoding>,<start line>,<end line>,<output>]

We need to focus on the parameters file, regexp and output:

• file = Path to the file to read
• regexp = Regular expression used to find the relevant data
• output = Returns data from matched groups

In this example, we need to retrieve “STAT” values inside our files. The content of our files is:

STAT	2
SPEED	100
PERF	480

Each file contains three lines. The data is is written as STRING<tab>VALUE.

To retrieve the VALUE next to STRING, this is the ‘vfs.file.regexp’ item prototype we’re going to use:

vfs.file.regexp[{#ITEM},^STRING\t(.*)$,,,,\1]

The first parameter is the file path of the file we wish to retrieve data from. ‘{#ITEM}’ will resolve to the absolute path of each file found with the ‘custom.discovery’ item.

The second parameter is a regular expression that starts at the beginning of the line, searches for ‘STRING’, then looks for a <tab> character and then puts anything it finds after that <tab> until the end of the line, a matched group.

The third parameters then tells the Zabbix Agent to retrieve the data found in the 1st matched group, which in this case, would be a value of ‘2’.

Instructions – Custom Discovery: Discover All the Things!

1. Add the file “userparameter_custom.conf” to your Zabbix Agent include directory.
   1. Be aware, the config “include” directory varies between operating systems:
      RHEL/CentOS: /etc/zabbix/zabbix_agentd.d/
      Debian/Ubuntu: /etc/zabbix/zabbix_agentd.conf.d/
      Windows: C:\zabbix\zabbix_agentd.conf.d\
2. Add these lines to the file, depending on your operating system:
   1. Linux (adjust /SOME/PATH to a directory of your choice)

      UserParameter=custom.discovery,l=$(ls /SOME/PATH/);echo -n '{"data":[';for i in ${l};do echo -n "{\"{#ITEM}\": \"$i\"},";done|sed -e 's:\},$:\}:';echo -n ']}';

   2. Windows (adjust C:\SOME\PATH to a directory of your choice)

      UserParameter=custom.discovery,powershell $l=ls 'C:\SOME\PATH';$o='{\"data\":[';foreach($i in $l){$o+=('{\"{#ITEM}\": \"'+$i.Name+'\"}');if($i -ne $l[-1]){$o+=','}};$o+=']}';echo $o;

3. Alternatively: You can add the custom UserParameter lines, at the bottom of your Zabbix Agent configuration file.
4. Restart the Zabbix Agent and test your new custom item keys with zabbix_get.
5. Create a new template and setup a Discovery rule:
   1. Type: Zabbix Agent
   2. Key: custom.discovery
   3. Keep lost resources period (in days): 0
      1. If you wish to retain data in Zabbix from deleted files, set this value higher than 0
6. Create an item prototype:
   1. Type: Zabbix Agent
   2. Key: vfs.file.regexp[{#ITEM},^STRING\t(.*)$,,,,\1]
      1. STRING should be the value you are looking for, in this case, STAT, SPEED or PERF.
   3. Type of information: Numeric (unsigned)
   4. Data type: Decimal
7. Create additional item prototypes for each type of value you wish to retrieve and edit the 2nd parameter of the ‘vfs.file.regexp’ key for each one.
8. Note: ‘Update interval’ is up to you, but do not set it too low if you have hundreds of items you wish to discover.
   If you are reading string values or values that can go negative, you have to adjust ‘Type of information’ and ‘Data type’ accordingly.
9. Add the template to relevant hosts.

That’s all there is to it! You’re now doing custom discovery of files as they appear inside your selected directory and reading values from them. As added or removed, Zabbix will create or remove items by itself!

Now, which files and what data are inside them, is entirely up to you. Many types of software create status files. You can also create a small script that pulls data from the software and populates one or more files with the information. Depending on how you end up formatting the files, you have to also edit your regular expression to find the values correctly.