Skip to content

Monitoring with Prometheus

Purpose

Prometheus with Node Exporter offers real-time monitoring by collecting detailed system metrics like CPU, memory, and disk usage. This deployment comes with pre-built Grafana dashboards, making monitoring metrics for monitored systems easy.

Prometheus is installed on the server, and Node Exporter is installed on the systems you wish to monitor.

This guide covers:

  • installing and configuring Prometheus on an Ubuntu 22.04 server
  • installing and configuring Node Exporter on Ubuntu 22.04 and FreeBSD systems

Requirements

To complete this guide, you'll need:

  • an Ubuntu 22.04 server system with a few free gigabytes of free space
  • at least one Linux or FreeBSD system to monitor
  • a web browser to view the Prometheus web interface
  • root access on all systems

Procedure

Server

Install Ubuntu 22.04 LTS server on a VM or physical gear.

Apply all updates.

Unless specified, run all commands as the root user (become root with: sudo -i).

Prometheus Install

Prometheus should run as a non-root user. Prometheus packages supplied by Ubuntu OS mirrors are usually out of date. Due to the minimal installation surface of the Prometheus server, this guide installs it from tarballs.

To update Prometheus when a new version is released, replace the binaries in /usr/local/sbin/ and restart the service.

First, create the user and group:

groupadd --system prometheus && \
useradd -s /sbin/nologin --system -g prometheus prometheus

Create the directories for Prometheus:

mkdir /etc/prometheus && \
mkdir /var/lib/prometheus

Download the latest Linux 2.x release from the Prometheus Download Page:

wget https://github.com/prometheus/prometheus/releases/download/v2.54.1/prometheus-2.54.1.linux-amd64.tar.gz

Unpack the tarball and move the prometheus binary to /usr/local/sbin/:

# unpack the tarball and cd into the directory it creates
tar zxvf rometheus-2.54.1.linux-amd64.tar.gz && \
cd prometheus-2.54.1.linux-amd64 && \
# move the binaries and set ownership
mv prometheus /usr/local/sbin && \
mv promtool /usr/local/sbin && \
chown prometheus:prometheus /usr/local/sbin/prometheus && \
chown prometheus:prometheus /usr/local/sbin/promtool

Copy the configuration files to /etc/prometheus/ and /var/lib/prometheus/:

mv consoles /etc/prometheus && \
mv console_libraries /etc/prometheus && \
mv prometheus.yml /etc/prometheus && \
chown prometheus:prometheus /etc/prometheus && \
chown -R prometheus:prometheus /etc/prometheus/consoles && \
chown -R prometheus:prometheus /etc/prometheus/console_libraries && \
chown -R prometheus:prometheus /var/lib/prometheus

Configure Prometheus

Edit the Prometheus configuration file at /etc/prometheus/prometheus.yml to contain:

# global config
global:
  scrape_interval:     15s  # Set the scrape interval to every 15 seconds.  Default is every 1 minute.
  evaluation_interval: 15s  # Evaluate rules every 15 seconds.  The default is every 1 minute.
                            # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# load rules once, periodically reload them according to the 'evaluation_interval' variable
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# what to monitor
scrape_configs:
  # monitor prometheus itself
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

  # remote systems running node_exporter to monitor, if it's not listed here it won't be in the web UI
  - job_name: 'node_exporter'
    static_configs:
    - targets: ['some.linux.host:9500']
    - targets: ['some.freebsd.host:9100']

Substitute your hosts to be monitored and ports for the fake values some.linux.host:9500 and some.freebsd.host:9100.

If unsure, leave the values for targets under static_config. They can easily be changed after the monitoring systems have Node Exporter running.

Systemd Unit

Create a Systemd Unit file in /etc/systemd/system/prometheus.service with the contents:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/sbin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Reload systemd to pick up the changes, enable the prometheus service, and start it:

systemctl daemon-reload && \
systemctl enable prometheus && \
systemctl start prometheus

Check the status of the running service:

systemctl status prometheus

UFW Firewall

If you are running the UFW firewall, allow ingress traffic on port tcp/9090 so you can access the UI with a browser:

ufw allow 9090/tcp

Web UI

You should now be able to log into the Prometheus Web UI at the IP address of your Prometheus system on port 9090.

Clients

Prometheus' Node Exporter is the client that gathers metrics on monitored systems and reports them to Prometheus.

This software must be installed and configured on each machine you want Prometheus to monitor.

To update Node Exporter when a new version is released, replace the binaries in /usr/local/sbin/ and restart the service.

Linux Node Exporter

Create the user and group that node_exporter will run as:

groupadd -g 99997 node_exporter && \
useradd node_exporter -u 99997 -g 99997 -m -s /bin/bash

Create the directory that node_exporter will use:

mkdir /var/lib/node_exporter && \
chown node_exporter:node_exporter /var/lib/node_exporter

Download and install the latest 2.x release of Node Exporter from the Prometheus Download page.

Unpack the tarball and move the binary into /usr/local/sbin/:

# unpack the tarball and cd into the directory it creates
tar zxvf node_exporter-1.8.2.linux-md64.tar.gz && \
cd node_exporter-1.8.2.linux-amd64 && \

# move the binary and set ownership
mv node_exporter /usr/local/sbin/ && \
chown root:root /usr/local/sbin/node_exporter

Install the systemd unit for Node Exporter by creating a file named /etc/systemd/system/node_exporter.service with the contents:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/sbin/node_exporter --web.listen-address=:9500 --collector.textfile.directory=/var/lib/node_exporter/
Restart=always
RestartSec=10s
NotifyAccess=all

[Install]
WantedBy=multi-user.target

Reload systemd, enable and start the service:

systemctl daemon-reload && \
systemctl enable node_exporter && \
systemctl restart node_exporter

Verify the service is running:

systemctl status node_exporter

FreeBSD Node Exporter

Install Node Exporter from the FreeBSD Package tree:

pkg update && \
pkg install node_exporter

Enable Node Exporter by adding this line to the /etc/rc.conf file:

node_exporter_enable="YES"

Start the node_exporter process and check it's status:

/usr/local/etc/rc.d/node_exporter restart && \
/usr/local/etc/rc.d/node_exporter status

Updating Monitored Targets

If you did not configure the target systems in your /etc/prometheus/prometheus.yml file before, do it now.

You can determine what port node_exporter is listening on by running:

netstat -an | grep LIST | grep node_exporter

Restart Prometheus if the config file has changed:

systemctl restart prometheus