Monitoring Services with Prometheus

Hi All! In this tutorial I will explain how we can monitor specific TCP ports where eventually our applications will be running.

Application we want to monitor — Postgres DB (Port — 5432)

Prerequisites -
1. We have a running Prometheus server installation
2. Make sure we have blackbox_exporter installed and tcp_connect module available.

# whereis blackbox_exporter
blackbox_exporter: /usr/local/bin/blackbox_exporter

Step 1 -
Update Prometheus configuration file prometheus.yml ->

- file_sd_configs:
  - files:
    - /etc/prometheus/file_sd/postgres-services.yml
metrics_path: /probe
params:
  module: [tcp_connect]
job_name: 'test-postgres'
tls_config:
  insecure_skip_verify: true
relabel_configs:
- source_labels:  [__address__]
  target_label: __param_target
- source_labels: [__param_target]
  target_label: instance
- replacement: 192.168.56.105:9115
  target_label: __address__
scrape_interval: 5s

Step 2 -
Create postgres-services.yml file which will have our postgres target systems information along with the desired labels.

# cat /etc/prometheus/file_sd/postgres-services.yml
- labels:
    project: postgres_hosts
  targets:
  - postgrestdb1.example.com:5432
  - postgrestdb2.example.com:5432

Step 3 -
Create an alert rule -

# cat /etc/prometheus/rules/rules.yml
- alert: PostgresTCPDown
  annotations:
    description: 'The automation service running at {{ $labels.instance }} has not been connectable for more than 2 minutes.'
    summary: '{{ $labels.instance }} is unreachable'
  expr: probe_success{job="test-5000"} == 0
    for: 2m
    labels:
    severity: 'critical'

Step 4 -
Update alertmanager configuration file as per your requirement.

For e.g here I will create a new entry for hosts running postgres db under receivers section.

- name: Postgres Hosts
  email_configs:
  - to: root@ansible-controller.example.com
    send_resolved: true
routes:
- match:
    project: postgres_hosts
  receiver: Postgres Hosts

Step 5 -
Reload the Prometheus and Alert Manager configurations.

# ps -ef | grep prometh
prometh+ 3765 1 0 Apr21 ? 00:47:21 /usr/local/bin/prometheus — config.file /etc/prometheus/prometheus.yml — storage.tsdb.path /var/lib/prometheus/ — web.console.templates=/etc/prometheus/consoles — web.console.libraries=/etc/prometheus/console_libraries — web.enable-admin-api — web.external-url=https://192.168.56.105:1234
# ps -ef | grep alertma
alertma+ 3226 1 0 Apr21 ? 00:06:19 /usr/local/bin/alertmanager — config.file=/etc/alertmanager/alertmanager.yml — storage.path=/data/alertmanager — web.external-url=https://192.168.56.105:3456
#kill -HUP 3226
#kill -HUP 3765

Step 6 -
Verify the functionality

If Postgres DB is down it will fire an alert -

Once you fix it the alerts will be gone!!
The same way using blackbox_exporter’s tcp_connect module you can monitor any service running on a tcp port.

Hope you like the tutorial. Please let me know your feedback in the response section.

Happy learning!