Prometheus and AlertManager step by step configuration with blackbox exporter on Ubuntu 18.04

Prometheus is a monitoring tool designed for recording real-time metrics in a time-series database. It is an open-source software project, written in Go. The Prometheus metrics are collected using HTTP pulls, allowing for higher performance and scalability.

Other tools which make Prometheus complete monitoring tool are:

• Exporters: These are libraries that help with exporting metrics from third-party systems as Prometheus metrics.
• PromQL: Prometheus query language which allows you to filter multi-dimensional time series data.

Install Prometheus

# apt-get update
# apt-get upgrade

Step 1: Create Prometheus system group

# groupadd --system prometheus

The group with ID < 1000 is a system group. Once the system group is added, create Prometheus system user and assign primary group create

# useradd -s /sbin/nologin --system -g prometheus prometheus

Step 2: Create data & configs directories for Prometheus
Prometheus needs a directory to store its data. We will create this under /var/lib/prometheus.

# mkdir /var/lib/prometheus

Prometheus primary configuration files directory is /etc/prometheus/. It will have some sub-directories:

# for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done

Step 3: Download Prometheus

# apt -y install wget curl vim
# mkdir -p /tmp/prometheus && cd /tmp/prometheus
# curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -
# tar xvf prometheus*.tar.gz
# cd prometheus*

Move the binary files to /usr/local/bin/ directory.

# mv prometheus promtool /usr/local/bin/
# prometheus --version
prometheus, version 2.19.2 (branch: HEAD, revision: c448ada63d83002e9c1d2c9f84e09f55a61f0ff7)
build user: root@dd72efe1549d
build date: 20200626–09:02:20
go version: go1.14.4
# promtool --version
promtool, version 2.19.2 (branch: HEAD, revision: c448ada63d83002e9c1d2c9f84e09f55a61f0ff7)
build user: root@dd72efe1549d
build date: 20200626–09:02:20
go version: go1.14.4

Move Prometheus configuration template to /etc directory.

#sudo mv prometheus.yml /etc/prometheus/prometheus.yml

Also move consoles and console_libraries to /etc/prometheus directory:

# mv consoles/ console_libraries/ /etc/prometheus/

Step 4: Configure Prometheus

root@labs-monitor-blr:/etc/ssl# cat /etc/prometheus/prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# — alertmanager:9093
# Load rules once and periodically evaluate them according to the global ‘evaluation_interval’.
rule_files:
# — “first_rules.yml”
# — “second_rules.yml”
# A scrape configuration containing exactly one endpoint to scrape:
# Here it’s Prometheus itself.

scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: ‘prometheus’

# metrics_path defaults to ‘/metrics’
# scheme defaults to ‘http’.

static_configs:
- targets: [‘localhost:9090’]

Create a Prometheus systemd Service unit file

root@labs-monitor-blr:~# tee /etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
— config.file=/etc/prometheus/prometheus.yml \
— storage.tsdb.path=/var/lib/prometheus \
— web.console.templates=/etc/prometheus/consoles \
— web.console.libraries=/etc/prometheus/console_libraries \
— web.listen-address=0.0.0.0:9090
— web.external-url=
SyslogIdentifier=prometheus
Restart=always
[Install]
WantedBy=multi-user.target
EOF

Change directory permissions.

root@labs-monitor-blr:~# for i in rules rules.d files_sd consoles console_libraries prometheus.yml; do sudo chown -R prometheus:prometheus /etc/prometheus/${i}; done
root@labs-monitor-blr:~# for i in rules rules.d files_sd consoles console_libraries; do sudo chmod -R 775 /etc/prometheus/${i}; done
root@labs-monitor-blr:~# chown -R prometheus:prometheus /var/lib/prometheus/

Reload systemd daemon and start the service:

root@labs-monitor-blr:~# systemctl daemon-reload
root@labs-monitor-blr:~# systemctl start prometheus
root@labs-monitor-blr:~# systemctl enable prometheus
root@labs-monitor-blr:~# systemctl status prometheus

If your server has a running firewall , you’ll need to open port 9090.

root@labs-monitor-blr:~# ufw allow 9090/tcp

Configure Nginx Proxy for Prometheus with SSL/TLS -

# apt-get install nginx
# cd /etc/nginx/sites-enabled/
# mv default labs-monitor-blr.conf
# cat labs-monitor-blr.conf
upstream labs-monitor-blr {
server localhost:9090;
}
server {
server_name labs-monitor-blr.eng.example.com;
listen 80;
access_log /var/log/nginx/sites/$server_name.http.log;
rewrite ^ https://$server_name$request_uri? permanent;
}
server {
server_name labs-monitor-blr.eng.example.com;
listen 443 ssl;
access_log /var/log/nginx/sites/$server_name.https.log;
ssl_certificate /etc/nginx/ssl/labs-monitor-blr.pem;
ssl_certificate_key /etc/nginx/ssl/labs-monitor-blr.key;
charset utf-8;
client_max_body_size 16m;
keepalive_timeout 70;
client_body_timeout 10;
client_header_timeout 10;
client_header_buffer_size 128;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://labs-monitor-blr;
}
}

# mkdir /etc/nginx/ssl/

The first step is to generate your self-signed certificate. To do this, log into your server and issue the following command:

#openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/labs-monitor-blr.key -out /etc/nginx/ssl/labs-monitor-blr.pem

You will be asked a few questions (such as country name, state, locality, etc.). The most important answer is the Common Name. For this question, answer with the server's IP Address

Once the command completes, the necessary files will be added to the /etc/nginx/ssl directory and are ready to use.

• -newkey rsa:4096: Create a 4096 bit RSA key for use with the certificate. RSA 2048 is the default on more recent versions of OpenSSL but to be sure of the key size, you should specify it during creation.
• -x509: Create a self-signed certificate.
• -sha256: Generate the certificate request using 265-bit SHA (Secure Hash Algorithm).
• -days: Determines the length of time in days that the certificate is being issued for. For a self-signed certificate, this value can be increased as necessary.
• -nodes: Create a certificate that does not require a passphrase. If this option is excluded, you will be required to enter the passphrase in the console each time the application using it is restarted.

root@labs-monitor-blr:~# nginx -t
root@labs-monitor-blr:/etc/nginx/ssl# ls
root@labs-monitor-blr:~# labs-monitor-blr.key labs-monitor-blr.pem
root@labs-monitor-blr:~# systemctl restart nginx
root@labs-monitor-blr:/etc/ssl# ufw app list
Available applications:
Nginx Full
Nginx HTTP
Nginx HTTPS
OpenSSH

Install and Configure Alert Manager

# curl -s https://api.github.com/repos/prometheus/alertmanager/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -
# tar xvzf alertmanager-0.21.0.linux-amd64.tar.gz
# ls
LICENSE NOTICE alertmanager alertmanager.yml amtool
# mv amtool alertmanager /usr/local/bin
# mkdir -p /etc/alertmanager
# mv alertmanager.yml /etc/alertmanager
# mkdir -p /data/alertmanager
# useradd -rs /bin/false alertmanager
# chown -R alertmanager:alertmanager /data/alertmanager /etc/alertmanager/*
# cd /lib/systemd/system
# touch alertmanager.service
# alertmanager -h
# vim alertmanager.service
# cat alertmanager.service

# systemctl enable alertmanager
Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager.service → /lib/systemd/system/alertmanager.service.
root@labs-monitor-blr:/lib/systemd/system# systemctl start alertmanager

Securing the AlertManager with TLS/SSL

#cd /etc/nginx/sites-available/
# cp labs-monitor-blr.conf labs-alertmanager-blr.conf
# cat labs-alertmanager-blr.conf
upstream labs-alertmanager-blr {
server localhost:9093;
}
server {
server_name labs-alertmanager-blr.eng.example.com;
listen 80;
access_log /var/log/nginx/sites/$server_name.http.log;
rewrite ^ https://$server_name$request_uri? permanent;
}
server {
server_name labs-alertmanager-blr.eng.example.com;
listen 443 ssl;
access_log /var/log/nginx/sites/$server_name.https.log;
ssl_certificate /etc/nginx/ssl/labs-monitor-blr.pem;
ssl_certificate_key /etc/nginx/ssl/labs-monitor-blr.key;
charset utf-8;
client_max_body_size 16m;
keepalive_timeout 70;
client_body_timeout 10;
client_header_timeout 10;
client_header_buffer_size 128;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://labs-alertmanager-blr;
}
}

#ln -s ../sites-available/labs-alertmanager-blr.conf .
#nginx -t
#systemctl restart nginx

Test both the services over UI ->
https://labs-monitor-blr.eng.example.com/targets
https://labs-alertmanager-blr.eng.example.com/#/alerts

Binding AlertManager with Prometheus

# cat /etc/prometheus/prometheus.yml

# cd /etc/prometheus/files_sd/
# cat alertmanager.yml
- targets:
- labs-alertmanager-blr.eng.example.com:9093

Install Blackbox exporter

The Blackbox exporter is a probing exporter used to monitor network endpoints such as HTTP, HTTPS, DNS, ICMP or TCP endpoints.

The Blackbox exporter provides metrics about HTTP latencies, DNS lookups latencies as well as statistics about SSL certs expiration.

The Blackbox exporter is mainly used to measure response times.

When running, the Blackbox exporter is going to expose a HTTP endpoint that can be used in order to monitor targets over the network. By default, the Blackbox exporter exposes the /probe endpoint that is used to retrieve those metrics.

For example, if my Blackbox exporter is running on port 9115, and if I query metrics for google.com, this is the endpoint that I can query from the exporter.

$ http://localhost:9115/probe?target=https://google.com&module=https_2xx

The main difference between the Blackbox exporter and application instrumenting is that the Blackbox exporter only focuses on availability while instrumentations can go more into details about performance.

Installing the Blackbox exporter for Prometheus

# cd /tmp/prometheus/
# curl -s https://api.github.com/repos/prometheus/blackbox_exporter/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -
# tar xvzf blackbox_exporter-0.17.0.linux-amd64.tar.gz
blackbox_exporter-0.17.0.linux-amd64/
blackbox_exporter-0.17.0.linux-amd64/blackbox_exporter
blackbox_exporter-0.17.0.linux-amd64/LICENSE
blackbox_exporter-0.17.0.linux-amd64/blackbox.yml
blackbox_exporter-0.17.0.linux-amd64/NOTICE
# cd blackbox_exporter-0.17.0.linux-amd64/
# mv blackbox_exporter /usr/local/bin
# mkdir -p /etc/blackbox
# mv blackbox.yml /etc/blackbox
# useradd -rs /bin/false blackbox
# chown blackbox:blackbox /usr/local/bin/blackbox_exporter
# chown -R blackbox:blackbox /etc/blackbox/*
# cd /lib/systemd/system
# touch blackbox.service
# cat blackbox.service

# systemctl enable blackbox.service
Created symlink /etc/systemd/system/multi-user.target.wants/blackbox.service → /lib/systemd/system/blackbox.service.
# systemctl start blackbox.service
# curl http://localhost:9115/metrics
root@labs-monitor-blr:/etc/prometheus/files_sd# cat blackbox.yml
- targets:
 - labs-monitor-blr.eng.example.com:9115

Binding BlackBox with Prometheus

# cat /etc/prometheus/prometheus.yml

# ps -ef | grep prom
prometh+ 55124 1 0 01:35 ? 00:00:00 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --web.listen-address=0.0.0.0:9090 --web.external-url=
root 55264 54499 0 01:41 pts/0 00:00:00 grep --color=auto prom
# kill -HUP 55124

Thats all for this tutorial.
To know more about How to create alert manager rules, monitoring diff service (TCP ports), monitoring windows hosts please follow the following tutorials ->

https://medium.com/@rakeshjain_17559/monitoring-tcp-services-with-prometheus-828a371f89b
https://medium.com/@rakeshjain_17559/prometheus-alerting-most-common-alert-rules-e9e219d4e949
https://medium.com/@rakeshjain_17559/monitoring-windows-servers-using-prometheus-wmi-exporter-eb082fcbaffb

Hope you like the tutorial. Please let me know your feedback in the response section.

Happy Learning!