Skip to content

Custom Discovery: Discover All the Things!

Custom DiscoveryHaving out-grown the included Zabbix Web Interface templates, I find myself more often than not, having to create new templates as well as entirely new ways of monitoring hardware and software. I often create custom discovery rules to dynamically create new items as services and devices change. This very powerful feature of Zabbix is called Low-level Discovery or LLD for short. The feature is described as follows by Zabbix:

Low-level discovery provides a way to automatically create items, triggers, and graphs for different entities on a computer. For instance, Zabbix can automatically start monitoring file systems or network interfaces on your machine, without the need to create items for each file system or network interface manually. Additionally it is possible to configure Zabbix to remove unneeded entities automatically based on actual results of periodically performed discovery.

With LLD, we are able to automatically, at timed intervals, discover new partitions, entire SNMP OID trees, system users, network interfaces and so on. The Zabbix Agent even has some built in items keys intended to be used with LLD. Item and Trigger prototypes then handles, based on the discovered data, creation of items and triggers.

For item discovery, Zabbix Agent and SNMP have their limits. Though with the agent and a little scripting, we can create our own Custom Discovery item keys!

For the purposes of this guide, we will try to discover all files within a directory and then via prototypes, create items that read information from these files.

Note: The ‘custom.discovery’ item can be edited to move through any array and doesn’t have to ‘ls’ a directory. One could use ‘Get-Service’ on Windows to discover services or ‘ls /sys/class/net’ on Linux to discover network interfaces.

The ‘vfs.file.regexp’ item would then of course have to be edited accordingly, to retrieve the information you’re interested in for each item.

Requirements

  • Zabbix 2.x and above
  • Zabbix Agent on the monitored host
  • Knowledge of Bash and PowerShell is a plus

How it works

Low-level Discovery is setup and the item key ‘custom.discovery’ will produce a JSON formatted list of files inside a directory of your choice with the tag ‘{#ITEM}’.

Using Item Prototypes, each ‘{#ITEM}’ object is created as items on the host(s)  with the built-in agent item key ‘vfs.file.regexp’.

In this example, ‘custom.discovery’ returns a list of absolute paths to files inside the directory ‘C:\SOME\PATH’ or ‘/SOME/PATH’. The path is hard-coded inside the Zabbix Agent configuration.

The key ‘vfs.file.regexp’ retrieves data from a file. Detailed information about the item key is available here.

vfs.file.regexp[file,regexp,<encoding>,<start line>,<end line>,<output>]

We need to focus on the parameters file, regexp and output:

  • file = Path to the file to read
  • regexp = Regular expression used to find the relevant data
  • output = Returns data from matched groups

In this example, we need to retrieve “STAT” values inside our files. The content of our files is:

STAT	2
SPEED	100
PERF	480

Each file contains three lines. The data is is written as STRING<tab>VALUE.

To retrieve the VALUE next to STRING, this is the ‘vfs.file.regexp’ item prototype we’re going to use:

vfs.file.regexp[{#ITEM},^STRING\t(.*)$,,,,\1]

The first parameter is the file path of the file we wish to retrieve data from. ‘{#ITEM}’ will resolve to the absolute path of each file found with the ‘custom.discovery’ item.

The second parameter is a regular expression that starts at the beginning of the line, searches for ‘STRING’, then looks for a <tab> character and then puts anything it finds after that <tab> until the end of the line, a matched group.

The third parameters then tells the Zabbix Agent to retrieve the data found in the 1st matched group, which in this case, would be a value of ‘2’.

Instructions – Custom Discovery: Discover All the Things!

  1. Add the file “userparameter_custom.conf” to your Zabbix Agent include directory.
    1. Be aware, the config “include” directory varies between operating systems:
      RHEL/CentOS: /etc/zabbix/zabbix_agentd.d/
      Debian/Ubuntu: /etc/zabbix/zabbix_agentd.conf.d/
      Windows: C:\zabbix\zabbix_agentd.conf.d\
  2. Add these lines to the file, depending on your operating system:
    1. Linux (adjust /SOME/PATH to a directory of your choice)
      UserParameter=custom.discovery,l=$(ls /SOME/PATH/);echo -n '{"data":[';for i in ${l};do echo -n "{\"{#ITEM}\": \"$i\"},";done|sed -e 's:\},$:\}:';echo -n ']}';
    2. Windows (adjust C:\SOME\PATH to a directory of your choice)
      UserParameter=custom.discovery,powershell $l=ls 'C:\SOME\PATH';$o='{\"data\":[';foreach($i in $l){$o+=('{\"{#ITEM}\": \"'+$i.Name+'\"}');if($i -ne $l[-1]){$o+=','}};$o+=']}';echo $o;
  3. Alternatively: You can add the custom UserParameter lines, at the bottom of your Zabbix Agent configuration file.
  4. Restart the Zabbix Agent and test your new custom item keys with zabbix_get.
  5. Create a new template and setup a Discovery rule:
    1. Type: Zabbix Agent
    2. Key: custom.discovery
    3. Keep lost resources period (in days): 0
      1. If you wish to retain data in Zabbix from deleted files, set this value higher than 0
  6. Create an item prototype:
    1. Type: Zabbix Agent
    2. Key: vfs.file.regexp[{#ITEM},^STRING\t(.*)$,,,,\1]
      1. STRING should be the value you are looking for, in this case, STAT, SPEED or PERF.
    3. Type of information: Numeric (unsigned)
    4. Data type: Decimal
  7. Create additional item prototypes for each type of value you wish to retrieve and edit the 2nd parameter of the ‘vfs.file.regexp’ key for each one.
  8. Note: ‘Update interval’ is up to you, but do not set it too low if you have hundreds of items you wish to discover.
    If you are reading string values or values that can go negative, you have to adjust ‘Type of information’ and ‘Data type’ accordingly.
  9. Add the template to relevant hosts.

That’s all there is to it! You’re now doing custom discovery of files as they appear inside your selected directory and reading values from them. As added or removed, Zabbix will create or remove items by itself!

Now, which files and what data are inside them, is entirely up to you. Many types of software create status files. You can also create a small script that pulls data from the software and populates one or more files with the information. Depending on how you end up formatting the files, you have to also edit your regular expression to find the values correctly.

7 Comments

  1. Sergey Sergey

    Is there is a way to make multiple triggers for 1 item?
    I want to make 1 item prototype and 1 trigger prototype. item prototype genetate 1 item but trigger prototype multiple triggers, this is idea.

    • Hi Sergey. If you discover and get for instance {#HOST}, you can simply create multiple prototype triggers using the discovered {#HOST} macro, let’s say three.

      So, when you discover, for each {#ITEM} you then create one item key and three triggers.

      Try adding multiple prototype triggers, and see what happens 🙂

  2. Mark Mark

    Hello

    Thanks for your sharing. This is a helpful guide.
    I have a question.
    Is there a way to set item prototype to update data 1 time only?

    I created a discovery rule to get a list of data file, and a item prototype to show content of file. As these data files are not updated , so I want the item prototype to collect and show them 1 time only.

    Please let me know your opinion about my case.

    Thanks a lot.
    Mark

    • Hi Mark,

      Items in Zabbix are meant to be updated on a schedule, but what you can do is set a very high interval between checks. In Zabbix 3.4.0 and above, you can also use Time suffixes. You could set the Update interval to “4w”, which would update the item about once a month, or even higher if you’d like.

      • Mark Mark

        Hi Martin

        Thanks for your reply.

        I tried to set the Update interval to “1d”, and the first update data is 1 day after item was created. Is there any way to show up data once item is created?

        Thanks a lot.
        Mark

        • Upon item discovery and creation, Zabbix schedules a check according to the chosen Update interval. The only way to currently circumvent this behaviour, is to create the items with a low Update interval, such as 1m or 10m, and then up the Update interval to the original desired value.

          Fortunately, a “Check now” feature for passive items, is scheduled for release with Zabbix 4.0. You can read more here: https://www.zabbix.com/documentation/4.0/manual/config/items/check_now

          • Mark Mark

            Thanks Martin a lot.

            Base on your great ideal, I set up the low Update interval to 1m, then create the script to check and update directly in zabbix database to 1d.
            Just a little inconvenience, sometime the item got data 2 times before my script run the db update.

            Again, many thanks for your helpful support.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.