Zabbix Agents are a key part of your monitoring infrastructure and a secure configuration ensures safe and reliable operation.
Excerpt from zabbix.com about agents:
A native Zabbix agent, developed in C language, may run on various supported platforms, including Linux, UNIX and Windows, and collect data such as CPU, memory, disk and network interface usage from a device.
They can do so much more though.
An agent, by default, keeps track of a huge assortment of metrics, like the ones mentioned above. It usually runs (at least under Linux) under the ‘zabbix’ user, with relatively restricted permissions. Additional item keys can be created using User Parameters with which you can create custom script snippets that gather data for you.
If the configuration option is enabled, agents are even capable of executing remote commands, given to them by the Zabbix Server!
We will go through a list of recommended settings as well as how to enable agent encryption with TLS PSK.
Secure Agent Configuration
The default agent configuration is relatively secure. Certain options do deserve a closer look though. Do also note, that the defaults listed below are daemon defaults. Values in shipped configuration files for your OS might differ.
- Server – List of comma delimited IP addresses, optionally in CIDR notation, or hostnames of Zabbix Servers. Incoming connections will be accepted only from the hosts listed here.
- Default: none
- Recommended: Should contain the IP addresses or host names of the Zabbix Servers or Proxies that should be able to query data from the agent.
- ListenIP – List of comma delimited IP addresses that the agent should listen on.
- Default: 0.0.0.0
- Recommended: If you utilize multiple IP addresses on your monitored hosts, it’s always a good idea to restrict listening to relevant addresses only.
- EnableRemoteCommands – Whether remote commands from Zabbix Server are allowed or not.
- Default: 0 (no)
- Recommended: 0 – However, if you do wish to use this feature you can enable it, but make sure your agents are properly secured and communication is encrypted!
Encrypted Communication
By default, agent communication is done in clear text. On private networks, this is usually not an issue. Regardless, the performance hit of the encryption scheme implemented by Zabbix is negligible, so using encryption is highly recommended.
For encryption, we can either choose RSA certificate-based encryption or PSK-based encryption. For the purposes of this post, PSK-based encryption is used.
PSK-based encryption is achieved by simply using a Pre-shared key. Each pre-shared key in Zabbix is actually a pair of:
- Non-secret PSK identity string – Do not put sensitive information in the PSK identity string – it is transmitted over the network unencrypted.
- Secret PSK string value – This string should be kept secret and secure!
The secret PSK string value can consist of between 32 and 512 hexadecimal digits. Below is an OpenSSL example for generating a pseudo-random 32 digit key:
$ openssl rand -hex 32
The generated key should be saved to a file. Make sure to avoid newlines. The file should only consist of a single line containing the pre-shared key.
This file should then be distributed to all hosts on which you wish to use agent encryption. Make sure to make the file readable only by the user under which the “zabbix_agentd” process runs. If someone has access to the pre-shared key, they are potentially able to decrypt agent communication. Securing this file is… key!
Agent
To enable encryption, the following settings should be adjusted:
- TLSConnect – How the agent should connect to server or proxy. Used for active checks.
- Default: unencrypted
- Recommended: psk
- TLSAccept – What incoming connections to accept, unencrypted, psk or cert, multiple values can be specified, separated by comma.
- Default: unencrypted
- Recommended: psk
- TLSPSKFile – Full path to a file containing the pre-shared key.
- Default: none
- Recommended: Path to file containing the pre-shared key. This file should only ever be readable by the user under which the “zabbix_agentd” process runs.
- TLSPSKIdentity – Unique, case sensitive string used to identify the pre-shared key.
- Default: none
- Recommended: Use a descriptive string value that identifies your pre-shared key, e.g. “AgentPSK001”.
Make sure to restart the agent, to apply settings.
Zabbix Web Frontend
Having just enabled agent encryption, your Zabbix Server and Proxies will no longer be able to communicate with your agents, since we explicitly removed the possibility of using unencrypted communication.
To configure host to use encryption, go to Configuration -> Hosts and select the hosts you wish to configure and then click “Mass update” at the bottom of the hosts list.
Select Encryption, put a check mark in Connections and fill out the necessary information
- Connections to host: PSK
- Connections from host: PSK
- PSK identity: The pre-shared key identity you chose before, e.g. “AgentPSK001”.
- PSK: The pre-shared key you generated earlier.
Click Update to apply. Your agent communication will resume, now fully encrypted.
Additionally
As with any other service, make sure you always firewall agents appropriately. For incoming traffic, the agent listens on port 10050/TCP and outgoing connections are made using remote port 10051/TCP.
Unknown bugs and exploits might be discovered, that could turn your agents against you! A simple, properly configured firewall is key to avoid unexpected surprises. A secure agent configuration should never overshadow basic security precautions.