SNMP Monitoring with Observium
Purpose
This guide describes setting up an SNMP-based monitoring system for non-Windows systems.
This involves configuring snmpd on systems to be monitored and installing Observium on a server to monitor the targets.
Any system that can run SNMPd is able to be monitored. This is not limited to UNIX-type systems and includes:
- switches
- routers
- some IoT devices
Traffic can be encrypted and authenticated between the systems, so it's safe for the monitoring traffic to traverse public networks.
Procedure
This guide describes how to install Net-SNMPd on systems that should be monitored.
Once there are valid targets to monitor, it will also guide you through installing, configuring, and adding targets to Observium.
Finally, configuration examples are shown that forward syslog messages to the Observium server.
NOTE: While you can configure this deployment using only IP addresses, it is recommended that you set up and configure an internal authoritative DNS resolver so your systems can be named. If you do not want to configure a DNS resolver, add entries for your target systems to the /etc/hosts file on the Observium server. Everything will work properly as long as Observium can resolve the name to an IP address.
Discovering SNMP-Enabled Devices
A scan of the ports snmpd listens on can reveal which devices on your network are already running snmpd and can be configured for monitoring. Use nmap to scan for daemons listening on udp/161 and enumerate some information:
Systems running an SNMP daemon will return data like:
Nmap scan report for myserver.secure.mynetwork.com (10.12.13.100)
Host is up (0.00016s latency).
PORT STATE SERVICE
161/udp open snmp
| snmp-info:
| enterprise: net-snmp
| engineIDFormat: unknown
| engineIDData: 54892938a12e1c6600000000
| snmpEngineBoots: 7
|_ snmpEngineTime: 88d20h42m19s
While these systems may respond to snmp queries, you will need to determine the SNMP version they use and the authentication settings.
For SNMPv2, you will need the 'community string'. SNMPv2 traffic cannot be encrypted.
For SNMPv3, you will need 'auth' username/password, the 'crypto' password, and the algorithm used by both. Log into your device's management interface to discover or set these parameters.
Target SNMPd Installation
The systems to be monitored need to run the SNMP daemon.
Traffic between Observium and the monitored systems will be encrypted using secret keys.
Linux
Installing net-snmpd on Linux systems is straightforward. Most popular distributions provide an in-tree package.
Use your system's package manager to install net-snmpd.
For Ubuntu type systems:
For RedHat type systems:
The configuration file for the software is in /etc/snmp/snmpd.conf. Make sure you keep the ownership as root:root and the file mode as 0600 to prevent anyone except for the root user from seeing the secrets used to secure the communications between Observium and the target.
FreeBSD
Use the Package system to install net-snmpd in FreeBSD:
Enable the snmp daemon in /etc/rc.conf:
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/etc/snmpd.conf"
snmptrapd_enable="NO"
The configuration file for the software is in /usr/local/etc/snmpd.conf. Make sure you keep the ownership as root:wheel and the file mode as 0640 to prevent anyone except for the root user from seeing the secrets used to secure the communications between Observium and the target.
Configure SNMPd
The snmp daemon comes with a default configuration file. Replace that with:
syslocation MyHomeNewtork
syscontact [email protected]
sysservices 79
rocommunity v2Password
rouser observer auth
createUser observer SHA supersecretkey AES supersecretkey2
disk /
In your configuration file, replace:
syslocationwith a string that describes your system's location. this can contain spacessyscontactwith your email address. no emails will be sent to this address from the daemonsupersecretkeywith some unique passphrase only you knowv2Passwordwith some unique passphrase only you know. this is used for snmpv2supersecretykey2with some other unique passphrase only you know
Restart the snmp daemon to pick up the changes.
On Linux, using systemd: systemctl restart snmpd
On FreeBSD: /usr/local/etc/rc.d/snmpd restart
Observium Installation
Now that you have some valid targets to monitor, install Observium inside an Ubuntu VM to monitor them all.
Set up an Ubuntu 22.04 LTS server system as the Observium server:
- 2 vCPUs
- 2GB RAM
- 64GB storage
Give the system a static IP address and configure the FQDN in the /etc/hosts file.
Install and configure snmpd on the Observium server so it can monitor itself. Do this before installing Observium.
Refer to the Debian/Ubuntu Installation instructions for Observium to complete the installation of the software on your system.
Configure Observium
The Observium instructions describe how to get rid of the "Welcome to your new Observium dashboard!" panel on the top of the page after logging into the web interface.
Adding New Hosts
Observium must be told what hosts to monitor.
To add a host to Observium, click on the 'Devices' drop-down at the top of the page, then click on the 'Add Device' button.
- fill in the hostname. if you cannot
pingthe hostname from the Observium server, the device addition will fail - set the 'Protocol Version' to 'v3'
- set the 'Auth Level' to 'authPriv'
- configure the 'Auth Username' and 'Auth Password' using the values you set in the
snmpd.conffile (supersecretkey) - set the 'Auth Algorithm' to 'SHA'
- configure the 'Crypto Password' using the value you set in the 'snmpd.conf' file (supersecretkey2)
- set the 'Crypto Algorithm' to 'AES (AES-128)'
Enable the 'Skip PING' option if the target system does not respond to ICMP echo requests.
Click on the 'Add device' button. Observium will walk the target device's snmp tree to discover the device on the next run. Observium polls every five minutes, so you should not have to wait long for stats and graphs to appear for the newly added device.
Forwarding Syslog
Syslog messages from target machines can be forwarded to Observium. Forwarded messages will appear as a section on each device's page in Observium.
Syslog on Observium
Configure rsyslogd on the Observium server to accept forwarded syslog messages from other systems.
Create a file named /etc/rsyslog.d/60-observium-server.conf with the contents:
module(load="imudp")
input(type="imudp"
port="514"
ruleset="observium")
module(load="omprog")
template(name="observium"
type="string"
string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg:::space-cc%||%programname%\n")
ruleset(name="observium") {
action(type="omprog"
binary="/opt/observium/syslog.php"
template="observium")
stop
}
Restart rsyslogd to pick up the changes:
Your Observium server should now accept forwarded messages from remote systems.
Syslog on Linux Targets
Configure rsyslogd on monitored targets to forward messages to the Observium system.
We only want to send WARN and ERROR class messages to Observium to prevent it from being spammed with meaningless messages.
Create a file on your Linux target that is already being monitored by Observium named /etc/rsyslog.d/50-observium.conf with the contents:
$PreserveFQDN on
*.warn @1012.13.15:514;RSYSLOG_SyslogProtocol23Format
*.error @10.12.13.15:514;RSYSLOG_SyslogProtocol23Format
Replace 10.12.13.15 with your Obserium system's IP address.
Restart rsyslogd to pick up the changes:
You should now see forwarded syslog messages on your monitor machine's page in Observium.
Troubleshooting
snmpd
On Linux systems, you can determine if the snmp daemon is running a few different ways.
Using systemd: systemctl status snmpd
Grepping syslogs: grep snmpd /var/log/syslog
Using netstat to check for a listening socket on udp/161:
On FreeBSD, this is a little different:
Using RC: /usr/local/etc/rc.d/snmpd status
Grepping syslogs: grep snmpd /var/log/messages
Using netstat to check for a listening socket on udp/161: