Integrate Elasticsearch 8 with Prometheus monitoring and Grafana dashboards

Update system packages

Start by updating your package manager to ensure you have the latest package information.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install Elasticsearch exporter

Download and install the Prometheus Elasticsearch exporter to collect metrics from your cluster.

cd /tmp
wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.7.0/elasticsearch_exporter-1.7.0.linux-amd64.tar.gz
tar -xzf elasticsearch_exporter-1.7.0.linux-amd64.tar.gz
sudo mv elasticsearch_exporter-1.7.0.linux-amd64/elasticsearch_exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/elasticsearch_exporter

Create exporter user and directories

Create a dedicated system user for the exporter service with minimal privileges.

sudo useradd --no-create-home --shell /bin/false elasticsearch_exporter
sudo mkdir -p /var/lib/elasticsearch_exporter
sudo chown elasticsearch_exporter:elasticsearch_exporter /var/lib/elasticsearch_exporter

Configure Elasticsearch exporter

Create the configuration file to specify Elasticsearch connection details and export settings.

elasticsearch:
  uri: "http://localhost:9200"
  username: ""
  password: ""
  timeout: 30s
  ssl_skip_verify: false

log:
  level: info
  format: json

metrics:
  cluster_info: true
  cluster_stats: true
  indices_stats: true
  shards_stats: true
  snapshots_stats: false

Create systemd service file

Configure the Elasticsearch exporter as a systemd service for automatic startup and management.

[Unit]
Description=Elasticsearch Exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=elasticsearch_exporter
Group=elasticsearch_exporter
ExecStart=/usr/local/bin/elasticsearch_exporter --config.file=/etc/elasticsearch_exporter/config.yml --web.listen-address=:9114
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Set proper permissions and start exporter

Apply correct ownership and permissions to configuration files, then enable the service.

sudo mkdir -p /etc/elasticsearch_exporter
sudo chown -R elasticsearch_exporter:elasticsearch_exporter /etc/elasticsearch_exporter
sudo chmod 755 /etc/elasticsearch_exporter
sudo chmod 644 /etc/elasticsearch_exporter/config.yml
sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch_exporter

Configure Prometheus scraping

Add the Elasticsearch exporter as a scraping target in your Prometheus configuration.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "elasticsearch_rules.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'elasticsearch'
    static_configs:
      - targets: ['localhost:9114']
    scrape_interval: 30s
    metrics_path: /metrics
    params:
      timeout: ['30s']

Create Elasticsearch alerting rules

Define Prometheus alerting rules to monitor critical Elasticsearch metrics and trigger notifications.

groups:
  - name: elasticsearch
    rules:
      - alert: ElasticsearchClusterRed
        expr: elasticsearch_cluster_health_status{color="red"} == 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Elasticsearch cluster status is RED"
          description: "Cluster {{ $labels.cluster }} health is RED"

      - alert: ElasticsearchClusterYellow
        expr: elasticsearch_cluster_health_status{color="yellow"} == 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Elasticsearch cluster status is YELLOW"
          description: "Cluster {{ $labels.cluster }} health is YELLOW for more than 10 minutes"

      - alert: ElasticsearchHighJVMMemory
        expr: elasticsearch_jvm_memory_used_bytes / elasticsearch_jvm_memory_max_bytes > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Elasticsearch JVM memory usage is high"
          description: "JVM memory usage is above 85% on node {{ $labels.name }}"

      - alert: ElasticsearchDiskSpaceLow
        expr: elasticsearch_filesystem_data_available_bytes / elasticsearch_filesystem_data_size_bytes < 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Elasticsearch disk space is low"
          description: "Less than 10% disk space available on node {{ $labels.name }}"

      - alert: ElasticsearchNodeDown
        expr: up{job="elasticsearch"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Elasticsearch exporter is down"
          description: "Elasticsearch exporter has been down for more than 2 minutes"

Restart Prometheus

Reload Prometheus configuration to apply the new scraping targets and alerting rules.

sudo systemctl restart prometheus
sudo systemctl status prometheus

Import Grafana dashboard

Add a comprehensive Elasticsearch dashboard to Grafana for visualizing cluster metrics and performance data.

curl -X POST \
  http://admin:admin@localhost:3000/api/dashboards/db \
  -H 'Content-Type: application/json' \
  -d '{
    "dashboard": {
      "id": null,
      "title": "Elasticsearch Cluster Monitoring",
      "tags": ["elasticsearch", "monitoring"],
      "timezone": "browser",
      "panels": [
        {
          "id": 1,
          "title": "Cluster Health Status",
          "type": "stat",
          "targets": [
            {
              "expr": "elasticsearch_cluster_health_status",
              "refId": "A"
            }
          ],
          "fieldConfig": {
            "defaults": {
              "mappings": [
                {
                  "options": {
                    "0": {
                      "text": "GREEN",
                      "color": "green"
                    },
                    "1": {
                      "text": "YELLOW",
                      "color": "yellow"
                    },
                    "2": {
                      "text": "RED",
                      "color": "red"
                    }
                  },
                  "type": "value"
                }
              ]
            }
          }
        }
      ],
      "time": {
        "from": "now-1h",
        "to": "now"
      },
      "refresh": "30s"
    }
  }'

Configure Grafana data source

Add Prometheus as a data source in Grafana to enable dashboard functionality.

curl -X POST \
  http://admin:admin@localhost:3000/api/datasources \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Prometheus",
    "type": "prometheus",
    "url": "http://localhost:9090",
    "access": "proxy",
    "isDefault": true
  }'

Set up Elasticsearch authentication

If your Elasticsearch cluster uses authentication, update the exporter configuration with credentials.

elasticsearch:
  uri: "https://localhost:9200"
  username: "monitoring_user"
  password: "secure_password"
  timeout: 30s
  ssl_skip_verify: false
  ca_cert: "/etc/elasticsearch/certs/ca.crt"
  client_cert: "/etc/elasticsearch/certs/client.crt"
  client_key: "/etc/elasticsearch/certs/client.key"

Create monitoring user in Elasticsearch

Create a dedicated user with minimal privileges for metrics collection.

curl -X POST "localhost:9200/_security/user/monitoring_user" \
  -H 'Content-Type: application/json' \
  -d '{
    "password" : "secure_password",
    "roles" : [ "monitoring_user" ],
    "full_name" : "Monitoring User",
    "email" : "monitoring@example.com"
  }'

Create comprehensive monitoring panels

Set up detailed Grafana panels for monitoring various Elasticsearch metrics including performance and resource utilization.

# Cluster Nodes Count
sum(elasticsearch_cluster_health_number_of_nodes)

Index Operations Rate
rate(elasticsearch_indices_indexing_index_total[5m])

Search Operations Rate
rate(elasticsearch_indices_search_query_total[5m])

JVM Memory Usage
elasticsearch_jvm_memory_used_bytes{area="heap"} / elasticsearch_jvm_memory_max_bytes{area="heap"} * 100

Disk Usage per Node
(elasticsearch_filesystem_data_size_bytes - elasticsearch_filesystem_data_available_bytes) / elasticsearch_filesystem_data_size_bytes * 100

Thread Pool Queue Size
elasticsearch_thread_pool_queue_count

GC Collection Time
rate(elasticsearch_jvm_gc_collection_seconds_sum[5m])