Setup Prometheus Blackbox Exporter for endpoint monitoring with SSL and alerting

Update system packages

Start by updating your package manager to ensure you have the latest security patches.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Create dedicated user for Blackbox Exporter

Running Blackbox Exporter as a dedicated user follows security best practices and isolates the service.

sudo useradd --no-create-home --shell /bin/false blackbox_exporter

Download and install Blackbox Exporter

Download the latest stable release and install it to the system binary directory.

cd /tmp
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.25.0/blackbox_exporter-0.25.0.linux-amd64.tar.gz
tar xzf blackbox_exporter-0.25.0.linux-amd64.tar.gz
sudo cp blackbox_exporter-0.25.0.linux-amd64/blackbox_exporter /usr/local/bin/
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter

Create configuration directory

Set up the configuration directory with proper permissions for the Blackbox Exporter user.

sudo mkdir -p /etc/blackbox_exporter
sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter

Configure Blackbox Exporter modules

Create the main configuration file with modules for HTTP, HTTPS, DNS, and ICMP monitoring.

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      tls_config:
        insecure_skip_verify: false
  https_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false
  http_post_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: POST
      headers:
        Content-Type: application/json
      body: '{}'
  tcp_connect:
    prober: tcp
    timeout: 5s
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      protocol: icmp
      preferred_ip_protocol: ip4
  dns:
    prober: dns
    timeout: 5s
    dns:
      query_name: "example.com"
      query_type: "A"
      valid_rcodes:
        - NOERROR
  ssl_expiry:
    prober: http
    timeout: 5s
    http:
      method: GET
      no_follow_redirects: true
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false

Set configuration file permissions

Secure the configuration file by setting appropriate ownership and permissions.

sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
sudo chmod 644 /etc/blackbox_exporter/blackbox.yml

Create systemd service file

Configure Blackbox Exporter to run as a systemd service with proper security constraints.

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/blackbox.yml --web.listen-address=:9115
Restart=always
RestartSec=3

Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/blackbox_exporter
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

[Install]
WantedBy=multi-user.target

Enable and start Blackbox Exporter

Start the service and enable it to run automatically on system boot.

sudo systemctl daemon-reload
sudo systemctl enable --now blackbox_exporter
sudo systemctl status blackbox_exporter

Configure firewall access

Allow Prometheus to access Blackbox Exporter on port 9115.

sudo ufw allow 9115/tcp comment 'Blackbox Exporter'

sudo firewall-cmd --permanent --add-port=9115/tcp
sudo firewall-cmd --reload

Add Blackbox Exporter to Prometheus

Configure Prometheus to scrape metrics from Blackbox Exporter and monitor your endpoints. Add this to your existing Prometheus configuration.

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://example.com
        - https://api.example.com
        - http://internal.example.com:8080
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  # SSL certificate monitoring
  - job_name: 'blackbox-ssl'
    metrics_path: /probe
    params:
      module: [ssl_expiry]
    static_configs:
      - targets:
        - https://example.com:443
        - https://secure.example.com:443
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  # ICMP ping monitoring
  - job_name: 'blackbox-icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - 8.8.8.8
        - 1.1.1.1
        - example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  # DNS monitoring
  - job_name: 'blackbox-dns'
    metrics_path: /probe
    params:
      module: [dns]
    static_configs:
      - targets:
        - 8.8.8.8:53
        - 1.1.1.1:53
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Reload Prometheus configuration

Apply the new configuration by reloading Prometheus.

sudo systemctl reload prometheus

Create Blackbox alerting rules

Set up alerting rules to notify you when endpoints become unavailable or SSL certificates are expiring.

groups:
name: blackbox
  rules:
  - alert: BlackboxProbeDown
    expr: probe_success == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Blackbox probe failed"
      description: "Probe {{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."

  - alert: BlackboxSlowProbe
    expr: probe_duration_seconds > 5
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Blackbox probe slow"
      description: "Probe {{ $labels.instance }} took {{ $value }} seconds to complete."

  - alert: BlackboxSslCertificateWillExpireSoon
    expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: "SSL certificate will expire soon"
      description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}"

  - alert: BlackboxSslCertificateWillExpireVerySoon
    expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "SSL certificate will expire very soon"
      description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}"

  - alert: BlackboxSslCertificateExpired
    expr: probe_ssl_earliest_cert_expiry - time() <= 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "SSL certificate expired"
      description: "SSL certificate for {{ $labels.instance }} has expired"

  - alert: BlackboxHttpStatusCode
    expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "HTTP Status Code"
      description: "HTTP status code for {{ $labels.instance }} is {{ $value }}"

  - alert: BlackboxProbeDnsLookupFailure
    expr: probe_dns_lookup_time_seconds < 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "DNS lookup failed"
      description: "DNS lookup for {{ $labels.instance }} failed"

Update Prometheus rules configuration

Add the new alerting rules to your Prometheus configuration file.

rule_files:
  - "rules/*.yml"

Validate and reload Prometheus

Check the configuration syntax and reload Prometheus to apply the new alerting rules.

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/blackbox.yml
sudo systemctl reload prometheus

Import Blackbox Exporter dashboard

Use the official Blackbox Exporter dashboard from Grafana Labs. Log into your Grafana instance and import dashboard ID 7587.

Key metrics for custom dashboards

Use these Prometheus queries to create custom dashboard panels for your specific monitoring needs.

# Probe success rate
probe_success

Response time
probe_duration_seconds

HTTP status codes
probe_http_status_code

SSL certificate expiry (days)
(probe_ssl_earliest_cert_expiry - time()) / 86400

DNS lookup time
probe_dns_lookup_time_seconds

HTTP response size
probe_http_content_length

TLS version
probe_tls_version_info

Custom HTTP headers and authentication

Add custom modules for endpoints requiring authentication or specific headers.

  http_basic_auth:
    prober: http
    timeout: 5s
    http:
      method: GET
      valid_status_codes: []
      basic_auth:
        username: "monitoring"
        password: "secure_password"
  
  http_custom_headers:
    prober: http
    timeout: 5s
    http:
      method: GET
      headers:
        User-Agent: "Blackbox-Exporter"
        X-API-Key: "your-api-key"
      valid_status_codes: [200, 201, 202]

TCP port monitoring

Monitor specific TCP services like databases or custom applications.

  tcp_connect_port:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
        - expect: "SSH-2.0-"
      tls: false
  
  mysql_connect:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
        - send: "SELECT 1"
        - expect: "1"
      tls: false