Setup Prometheus Blackbox Exporter SSL Monitoring

Configure Prometheus Blackbox Exporter to monitor HTTP endpoints, SSL certificates, and DNS resolution with automated alerting rules and Grafana dashboards.

Prerequisites

Prometheus server installed and configured
Grafana dashboard access
Root or sudo access
Network access to monitored endpoints

What this solves

Prometheus Blackbox Exporter monitors your services from the outside, checking if websites respond, SSL certificates are valid, and DNS resolves correctly. This external monitoring catches issues your internal monitoring might miss, like network connectivity problems or certificate expiration.

Step-by-step installation

Update system packages

Start by updating your package manager to ensure you have the latest security patches.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Create dedicated user for Blackbox Exporter

Running Blackbox Exporter as a dedicated user follows security best practices and isolates the service.

sudo useradd --no-create-home --shell /bin/false blackbox_exporter

Download and install Blackbox Exporter

Download the latest stable release and install it to the system binary directory.

cd /tmp
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.25.0/blackbox_exporter-0.25.0.linux-amd64.tar.gz
tar xzf blackbox_exporter-0.25.0.linux-amd64.tar.gz
sudo cp blackbox_exporter-0.25.0.linux-amd64/blackbox_exporter /usr/local/bin/
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter

Create configuration directory

Set up the configuration directory with proper permissions for the Blackbox Exporter user.

sudo mkdir -p /etc/blackbox_exporter
sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter

Configure Blackbox Exporter modules

Create the main configuration file with modules for HTTP, HTTPS, DNS, and ICMP monitoring.

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      tls_config:
        insecure_skip_verify: false
  https_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false
  http_post_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: POST
      headers:
        Content-Type: application/json
      body: '{}'
  tcp_connect:
    prober: tcp
    timeout: 5s
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      protocol: icmp
      preferred_ip_protocol: ip4
  dns:
    prober: dns
    timeout: 5s
    dns:
      query_name: "example.com"
      query_type: "A"
      valid_rcodes:
        - NOERROR
  ssl_expiry:
    prober: http
    timeout: 5s
    http:
      method: GET
      no_follow_redirects: true
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false

Set configuration file permissions

Secure the configuration file by setting appropriate ownership and permissions.

sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
sudo chmod 644 /etc/blackbox_exporter/blackbox.yml

Create systemd service file

Configure Blackbox Exporter to run as a systemd service with proper security constraints.

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/blackbox.yml --web.listen-address=:9115
Restart=always
RestartSec=3

Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/blackbox_exporter
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

[Install]
WantedBy=multi-user.target

Enable and start Blackbox Exporter

Start the service and enable it to run automatically on system boot.

sudo systemctl daemon-reload
sudo systemctl enable --now blackbox_exporter
sudo systemctl status blackbox_exporter

Configure firewall access

Allow Prometheus to access Blackbox Exporter on port 9115.

sudo ufw allow 9115/tcp comment 'Blackbox Exporter'

sudo firewall-cmd --permanent --add-port=9115/tcp
sudo firewall-cmd --reload

Configure Prometheus integration

Add Blackbox Exporter to Prometheus

Configure Prometheus to scrape metrics from Blackbox Exporter and monitor your endpoints. Add this to your existing Prometheus configuration.

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://example.com
        - https://api.example.com
        - http://internal.example.com:8080
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  # SSL certificate monitoring
  - job_name: 'blackbox-ssl'
    metrics_path: /probe
    params:
      module: [ssl_expiry]
    static_configs:
      - targets:
        - https://example.com:443
        - https://secure.example.com:443
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  # ICMP ping monitoring
  - job_name: 'blackbox-icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - 8.8.8.8
        - 1.1.1.1
        - example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  # DNS monitoring
  - job_name: 'blackbox-dns'
    metrics_path: /probe
    params:
      module: [dns]
    static_configs:
      - targets:
        - 8.8.8.8:53
        - 1.1.1.1:53
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Reload Prometheus configuration

Apply the new configuration by reloading Prometheus.

sudo systemctl reload prometheus

Configure alerting rules

Create Blackbox alerting rules

Set up alerting rules to notify you when endpoints become unavailable or SSL certificates are expiring.

groups:
name: blackbox  rules:
  - alert: BlackboxProbeDown
    expr: probe_success == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Blackbox probe failed"
      description: "Probe {{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."

  - alert: BlackboxSlowProbe
    expr: probe_duration_seconds > 5
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Blackbox probe slow"
      description: "Probe {{ $labels.instance }} took {{ $value }} seconds to complete."

  - alert: BlackboxSslCertificateWillExpireSoon
    expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: "SSL certificate will expire soon"
      description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}"

  - alert: BlackboxSslCertificateWillExpireVerySoon
    expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "SSL certificate will expire very soon"
      description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}"

  - alert: BlackboxSslCertificateExpired
    expr: probe_ssl_earliest_cert_expiry - time() <= 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "SSL certificate expired"
      description: "SSL certificate for {{ $labels.instance }} has expired"

  - alert: BlackboxHttpStatusCode
    expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "HTTP Status Code"
      description: "HTTP status code for {{ $labels.instance }} is {{ $value }}"

  - alert: BlackboxProbeDnsLookupFailure
    expr: probe_dns_lookup_time_seconds < 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "DNS lookup failed"
      description: "DNS lookup for {{ $labels.instance }} failed"

Update Prometheus rules configuration

Add the new alerting rules to your Prometheus configuration file.

rule_files:
  - "rules/*.yml"

Validate and reload Prometheus

Check the configuration syntax and reload Prometheus to apply the new alerting rules.

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/blackbox.yml
sudo systemctl reload prometheus

Configure Grafana dashboards

Import Blackbox Exporter dashboard

Use the official Blackbox Exporter dashboard from Grafana Labs. Log into your Grafana instance and import dashboard ID 7587.

Dashboard ID: Import dashboard 7587 for comprehensive Blackbox Exporter monitoring, or create custom panels using the queries below.

Key metrics for custom dashboards

Use these Prometheus queries to create custom dashboard panels for your specific monitoring needs.

# Probe success rate
probe_success

Response time
probe_duration_seconds

HTTP status codes
probe_http_status_code

SSL certificate expiry (days)
(probe_ssl_earliest_cert_expiry - time()) / 86400

DNS lookup time
probe_dns_lookup_time_seconds

HTTP response size
probe_http_content_length

TLS version
probe_tls_version_info

Advanced configuration options

Custom HTTP headers and authentication

Add custom modules for endpoints requiring authentication or specific headers.

  http_basic_auth:
    prober: http
    timeout: 5s
    http:
      method: GET
      valid_status_codes: []
      basic_auth:
        username: "monitoring"
        password: "secure_password"
  
  http_custom_headers:
    prober: http
    timeout: 5s
    http:
      method: GET
      headers:
        User-Agent: "Blackbox-Exporter"
        X-API-Key: "your-api-key"
      valid_status_codes: [200, 201, 202]

TCP port monitoring

Monitor specific TCP services like databases or custom applications.

  tcp_connect_port:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
        - expect: "SSH-2.0-"
      tls: false
  
  mysql_connect:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
        - send: "SELECT 1"
        - expect: "1"
      tls: false

Security note: Store sensitive credentials like API keys and passwords in external files or use Prometheus's file-based service discovery to keep them out of the main configuration.

Verify your setup

# Check Blackbox Exporter status
sudo systemctl status blackbox_exporter

Test the web interface
curl http://localhost:9115/metrics

Test HTTP probe manually
curl "http://localhost:9115/probe?target=https://example.com&module=http_2xx"

Check SSL certificate monitoring
curl "http://localhost:9115/probe?target=https://example.com&module=ssl_expiry" | grep probe_ssl_earliest_cert_expiry

Verify Prometheus is scraping targets
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.job | contains("blackbox"))'

Check alerting rules are loaded
curl http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name=="blackbox")'

Test connectivity to monitored endpoints
ping -c 3 example.com
nslookup example.com

Common issues

Symptom	Cause	Fix
Blackbox Exporter won't start	Configuration syntax error	`sudo journalctl -u blackbox_exporter -f` and check YAML syntax
Probes timing out	Network connectivity or DNS issues	Test manually with `curl` and check firewall rules
SSL probes failing	Certificate validation or TLS version mismatch	Check certificate with `openssl s_client -connect example.com:443`
Prometheus not scraping Blackbox	Incorrect relabel configuration	Verify target labels in Prometheus web UI under Status > Targets
DNS probes not working	Incorrect DNS server or query configuration	Test with `dig @8.8.8.8 example.com`
ICMP probes require root	Blackbox Exporter needs CAP_NET_RAW capability	`sudo setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter`

Next steps

Running this in production?

Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European teams.

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash

set -euo pipefail

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Configuration
BLACKBOX_VERSION="0.25.0"
BLACKBOX_USER="blackbox_exporter"
BLACKBOX_GROUP="blackbox_exporter"
BLACKBOX_PORT="9115"
CONFIG_DIR="/etc/blackbox_exporter"
CONFIG_FILE="$CONFIG_DIR/blackbox.yml"
BINARY_PATH="/usr/local/bin/blackbox_exporter"
SERVICE_FILE="/etc/systemd/system/blackbox_exporter.service"

# Cleanup function
cleanup() {
    if [ $? -ne 0 ]; then
        echo -e "${RED}[ERROR] Installation failed. Cleaning up...${NC}"
        systemctl stop blackbox_exporter 2>/dev/null || true
        systemctl disable blackbox_exporter 2>/dev/null || true
        rm -f "$SERVICE_FILE" "$BINARY_PATH" 2>/dev/null || true
        rm -rf "$CONFIG_DIR" 2>/dev/null || true
        userdel "$BLACKBOX_USER" 2>/dev/null || true
    fi
}
trap cleanup ERR

print_success() {
    echo -e "${GREEN}$1${NC}"
}

print_error() {
    echo -e "${RED}$1${NC}"
}

print_warning() {
    echo -e "${YELLOW}$1${NC}"
}

# Check if running as root
if [ "$EUID" -ne 0 ]; then
    print_error "Please run this script as root or with sudo"
    exit 1
fi

# Auto-detect distribution
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian)
            PKG_MGR="apt"
            PKG_UPDATE="apt update && apt upgrade -y"
            PKG_INSTALL="apt install -y"
            ;;
        almalinux|rocky|centos|rhel|ol|fedora)
            PKG_MGR="dnf"
            PKG_UPDATE="dnf update -y"
            PKG_INSTALL="dnf install -y"
            ;;
        amzn)
            PKG_MGR="yum"
            PKG_UPDATE="yum update -y"
            PKG_INSTALL="yum install -y"
            ;;
        *)
            print_error "Unsupported distribution: $ID"
            exit 1
            ;;
    esac
else
    print_error "Cannot detect distribution. /etc/os-release not found."
    exit 1
fi

echo -e "${GREEN}Installing Prometheus Blackbox Exporter${NC}"
echo "Distribution: $PRETTY_NAME"
echo "Package Manager: $PKG_MGR"
echo

# Step 1: Update system packages
echo "[1/8] Updating system packages..."
eval "$PKG_UPDATE"
$PKG_INSTALL wget tar curl
print_success "✓ System packages updated"

# Step 2: Create dedicated user
echo "[2/8] Creating dedicated user for Blackbox Exporter..."
if ! id "$BLACKBOX_USER" &>/dev/null; then
    useradd --no-create-home --shell /bin/false "$BLACKBOX_USER"
    print_success "✓ User $BLACKBOX_USER created"
else
    print_warning "! User $BLACKBOX_USER already exists"
fi

# Step 3: Download and install Blackbox Exporter
echo "[3/8] Downloading and installing Blackbox Exporter..."
cd /tmp
wget -q "https://github.com/prometheus/blackbox_exporter/releases/download/v${BLACKBOX_VERSION}/blackbox_exporter-${BLACKBOX_VERSION}.linux-amd64.tar.gz"
tar xzf "blackbox_exporter-${BLACKBOX_VERSION}.linux-amd64.tar.gz"
cp "blackbox_exporter-${BLACKBOX_VERSION}.linux-amd64/blackbox_exporter" "$BINARY_PATH"
chown "$BLACKBOX_USER:$BLACKBOX_GROUP" "$BINARY_PATH"
chmod 755 "$BINARY_PATH"
rm -rf "blackbox_exporter-${BLACKBOX_VERSION}"*
print_success "✓ Blackbox Exporter installed to $BINARY_PATH"

# Step 4: Create configuration directory
echo "[4/8] Creating configuration directory..."
mkdir -p "$CONFIG_DIR"
chown "$BLACKBOX_USER:$BLACKBOX_GROUP" "$CONFIG_DIR"
chmod 755 "$CONFIG_DIR"
print_success "✓ Configuration directory created"

# Step 5: Create configuration file
echo "[5/8] Creating Blackbox Exporter configuration..."
cat > "$CONFIG_FILE" << 'EOF'
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      tls_config:
        insecure_skip_verify: false
  https_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false
  http_post_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: POST
      headers:
        Content-Type: application/json
      body: '{}'
  tcp_connect:
    prober: tcp
    timeout: 5s
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      protocol: icmp
      preferred_ip_protocol: ip4
  dns:
    prober: dns
    timeout: 5s
    dns:
      query_name: "example.com"
      query_type: "A"
      valid_rcodes:
        - NOERROR
  ssl_expiry:
    prober: http
    timeout: 5s
    http:
      method: GET
      no_follow_redirects: true
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false
EOF

chown "$BLACKBOX_USER:$BLACKBOX_GROUP" "$CONFIG_FILE"
chmod 644 "$CONFIG_FILE"
print_success "✓ Configuration file created"

# Step 6: Create systemd service file
echo "[6/8] Creating systemd service..."
cat > "$SERVICE_FILE" << EOF
[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=$BLACKBOX_USER
Group=$BLACKBOX_GROUP
Type=simple
ExecStart=$BINARY_PATH --config.file=$CONFIG_FILE --web.listen-address=:$BLACKBOX_PORT
Restart=always
RestartSec=3

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

[Install]
WantedBy=multi-user.target
EOF

chmod 644 "$SERVICE_FILE"
print_success "✓ Systemd service file created"

# Step 7: Enable and start service
echo "[7/8] Enabling and starting Blackbox Exporter service..."
systemctl daemon-reload
systemctl enable blackbox_exporter
systemctl start blackbox_exporter

# Wait for service to start
sleep 3
print_success "✓ Blackbox Exporter service started"

# Step 8: Configure firewall
echo "[8/8] Configuring firewall..."
if command -v ufw &> /dev/null; then
    ufw allow "$BLACKBOX_PORT"/tcp
    print_success "✓ UFW firewall rule added"
elif command -v firewall-cmd &> /dev/null; then
    firewall-cmd --permanent --add-port="$BLACKBOX_PORT"/tcp
    firewall-cmd --reload
    print_success "✓ Firewalld rule added"
else
    print_warning "! No supported firewall found. Please manually open port $BLACKBOX_PORT"
fi

# Verification
echo
echo "Verifying installation..."
if systemctl is-active --quiet blackbox_exporter; then
    print_success "✓ Blackbox Exporter service is running"
else
    print_error "✗ Blackbox Exporter service is not running"
    exit 1
fi

if curl -s "localhost:$BLACKBOX_PORT/metrics" > /dev/null; then
    print_success "✓ Blackbox Exporter is responding on port $BLACKBOX_PORT"
else
    print_error "✗ Blackbox Exporter is not responding"
    exit 1
fi

echo
print_success "🎉 Blackbox Exporter installation completed successfully!"
echo
echo "Service status: $(systemctl is-active blackbox_exporter)"
echo "Metrics endpoint: http://$(hostname -I | awk '{print $1}'):$BLACKBOX_PORT/metrics"
echo "Configuration file: $CONFIG_FILE"
echo
echo "Example probe URL:"
echo "  http://localhost:$BLACKBOX_PORT/probe?module=http_2xx&target=http://example.com"
echo
echo "To view logs: journalctl -u blackbox_exporter -f"

Review the script before running. Execute with: bash install.sh

#prometheus #blackbox-exporter #monitoring #ssl-monitoring #endpoint-monitoring

Setup Prometheus Blackbox Exporter for endpoint monitoring with SSL and alerting

Prerequisites

What this solves

Step-by-step installation

Update system packages

Create dedicated user for Blackbox Exporter

Download and install Blackbox Exporter

Create configuration directory

Configure Blackbox Exporter modules

Set configuration file permissions

Create systemd service file

Security settings

Enable and start Blackbox Exporter

Configure firewall access

Configure Prometheus integration

Add Blackbox Exporter to Prometheus

Reload Prometheus configuration

Configure alerting rules

Create Blackbox alerting rules

Update Prometheus rules configuration

Validate and reload Prometheus

Configure Grafana dashboards

Import Blackbox Exporter dashboard

Key metrics for custom dashboards

Response time

HTTP status codes

SSL certificate expiry (days)

DNS lookup time

HTTP response size

TLS version

Advanced configuration options

Custom HTTP headers and authentication

TCP port monitoring

Verify your setup

Test the web interface

Test HTTP probe manually

Check SSL certificate monitoring

Verify Prometheus is scraping targets

Check alerting rules are loaded

Test connectivity to monitored endpoints

Common issues

Next steps

Running this in production?

Related tutorials

Configure Prometheus alerting with AlertManager notifications and webhook integration

Set up Prometheus and Grafana monitoring stack with Docker Compose

Set up Telegraf custom plugins for application monitoring with Prometheus and InfluxDB integration

Don't want to manage this yourself?