Monitor network services automatically

It is extremely important to monitor network services and their connectivity. At any given time, there’s hundreds, if not thousands, of services running on your servers. Many of those have to be reachable over a network. Web services need to be available typically on port 80 and/or 443, FTP services on port 21, MySQL on 3306 and so on.

Keeping up with “what service is running where and on what port” is very time consuming. Keeping an eye on those services actually being reachable, is a humongous task. As your environment grows, it can eventually become almost impossible to stay on top of.

With the help of Zabbix LLD (Low-level Discovery), the awesome Nmap tool and a little bash scripting, we’ll be able to routinely scan our hosts for running network services. LLD will also take care of dynamically adding Simple checks with accompanying triggers.

Requirements

• Nmap installed on Zabbix Server and Proxies (https://nmap.org)
  • Redhat/CentOS: yum install nmap
  • Ubuntu/Debian: apt-get install nmap

How it works

Using LLD, an External check is performed, which makes your Zabbix Server or Proxy execute a script (tcp-discovery.sh). The script fires up Nmap, which scans the host (from the server or proxy, remotely) and returns all open ports. It will also return their protocol and what service is running behind the port. LLD makes sure items and triggers for all ports are created, activated and ready to monitor network services.

Disclaimer

Nmap is an extremely useful tool. Unfortunately it cannot actually perform real magic. If services are using non-standard ports, the service might be incorrectly named upon discovery. Be aware of this!

The script portion of this template is loosely based on work done by: https://meinit.nl/zabbix-low-level-discovery-tcp-ports-host

Instructions – Monitor network services automatically

1. First, head over to Zabbix Share to fetch the template and script.
2. Copy “tcp-discovery.sh” to your Zabbix Servers and Proxies and place it in “/usr/local/share/zabbix/externalscripts”*
   1. *Check your server and proxy configuration for the correct folder, look for the tag “ExternalScripts”
3. Make the script executable: chmod +x /usr/local/share/zabbix/externalscripts/tcp-discovery.sh
4. Import the template and assign it to your host(s).

If you run into trouble executing the script, here’s an excerpt from the Zabbix Wiki:

Zabbix server will look in the directory defined as the location for external scripts (parameter ‘ExternalScripts’ in Zabbix server configuration file) and execute the command. The command will be executed as the user Zabbix server runs as, so any access permissions or environment variables should be handled in a wrapper script, if necessary, and permissions on the command should allow that user to execute it. Only commands in the specified directory are available for execution.

Script: tcp-discovery.sh

#!/bin/bash
usage(){ echo "Usage: ./$(basename $0) hostname. For example: ./$(basename $0) server1.example.com";}
if [ -z $1 ]; then usage; exit 1; fi
IFS=$'\n'
LIST=$(nmap -T5 -F ${1} | grep 'open')
echo -n '{"data":['
for s in $LIST; do
        PORT=$(echo $s | cut -d/ -f1)
        PROTO=$(echo $s | cut -d/ -f2 | awk '{sub(/[[:space:]].*/,""); print}')
        SERVICE=$(echo $s | awk '{print $3}')
        echo -n '{"{#PORT}":"'${PORT}'","{#PROTO}":"'${PROTO}'","{#SERVICE}":"'${SERVICE}'"},'
done |sed -e 's:\},$:\}:'
echo -n ']}'
unset IFS