Set up Kibana alerting and monitoring with Elasticsearch Watcher for automated threat detection

Update system packages

Start by updating your system packages to ensure you have the latest security patches and package definitions.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install required dependencies

Install Java, curl, and other dependencies needed for Elasticsearch and Kibana operations.

sudo apt install -y openjdk-11-jdk curl wget gnupg apt-transport-https

sudo dnf install -y java-11-openjdk curl wget gnupg

Add Elasticsearch repository

Add the official Elasticsearch repository to install the latest stable version with Watcher support.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
sudo tee /etc/yum.repos.d/elasticsearch.repo << 'EOF'
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF

Install Elasticsearch and Kibana

Install Elasticsearch with Watcher capabilities and Kibana for the web interface and alerting configuration.

sudo apt install -y elasticsearch kibana

sudo dnf --enablerepo=elasticsearch install -y elasticsearch kibana

Configure Elasticsearch for Watcher

Configure Elasticsearch with proper memory settings and enable Watcher for alerting functionality.

cluster.name: security-monitoring
node.name: watcher-node-1
network.host: localhost
http.port: 9200
discovery.type: single-node

Security settings
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl.enabled: false
xpack.security.transport.ssl.enabled: false

Watcher settings
xpack.watcher.enabled: true
xpack.notification.email.default_account: gmail_account

Set JVM heap size for Elasticsearch

Configure JVM memory allocation for optimal Elasticsearch performance. Use half of available RAM, up to 32GB maximum.

-Xms2g
-Xmx2g

Configure Kibana settings

Configure Kibana to connect to Elasticsearch and enable alerting features with proper network settings.

server.port: 5601
server.host: "0.0.0.0"
server.name: "kibana-alerting"
elasticsearch.hosts: ["http://localhost:9200"]

Security settings
xpack.security.enabled: true
xpack.encryptedSavedObjects.encryptionKey: "a7a6311933d3503b89bc2dbc36572c33a6c10925682e591bffcab6911c06786d"
xpack.reporting.encryptionKey: "a7a6311933d3503b89bc2dbc36572c33a6c10925682e591bffcab6911c06786d"
xpack.security.encryptionKey: "a7a6311933d3503b89bc2dbc36572c33a6c10925682e591bffcab6911c06786d"

Alerting settings
xpack.actions.enabled: true
xpack.alerting.enabled: true

Set up email notification configuration

Configure email settings for Watcher notifications. This example uses Gmail SMTP with app-specific passwords.

# Email notification settings
xpack.notification.email.account:
  gmail_account:
    profile: gmail
    smtp:
      auth: true
      starttls.enable: true
      host: smtp.gmail.com
      port: 587
      user: "your-email@gmail.com"
      password: "your-app-specific-password"

Create directory structure and set permissions

Create necessary directories and set proper ownership for Elasticsearch and Kibana services.

sudo mkdir -p /var/log/elasticsearch /var/log/kibana /etc/kibana/certs
sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch /var/lib/elasticsearch /var/log/elasticsearch
sudo chown -R kibana:kibana /etc/kibana /var/log/kibana
sudo chmod 750 /etc/elasticsearch /etc/kibana
sudo chmod 640 /etc/elasticsearch/elasticsearch.yml /etc/kibana/kibana.yml

Start and enable services

Start Elasticsearch first, then Kibana, and enable them to start automatically on boot.

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
sudo systemctl status elasticsearch

Wait for Elasticsearch to start, then start Kibana
sleep 30
sudo systemctl enable kibana
sudo systemctl start kibana
sudo systemctl status kibana

Configure built-in users and passwords

Set up the built-in Elasticsearch users with secure passwords for authentication.

sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

Create Slack webhook integration

Configure Slack webhook for notifications. First, create an incoming webhook in your Slack workspace, then add the configuration.

# Slack notification settings
xpack.notification.slack:
  account:
    monitoring:
      url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"

Create a sample threat detection watcher

Create a watcher that monitors for failed SSH login attempts and sends alerts when thresholds are exceeded.

curl -X PUT "localhost:9200/_watcher/watch/ssh_brute_force_detector" -H 'Content-Type: application/json' -u elastic:your-password -d'
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": ["auth-*"],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "message": "Failed password"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "failed_logins_by_ip": {
              "terms": {
                "field": "source_ip.keyword",
                "size": 10
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 5
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "gmail",
        "to": ["security@example.com"],
        "subject": "SSH Brute Force Attack Detected",
        "body": "Multiple failed SSH login attempts detected. Total attempts: {{ctx.payload.hits.total}} in the last 5 minutes."
      }
    },
    "send_slack": {
      "slack": {
        "account": "monitoring",
        "message": {
          "to": ["#security-alerts"],
          "text": "🚨 SSH Brute Force Attack Detected: {{ctx.payload.hits.total}} failed attempts in 5 minutes"
        }
      }
    }
  }
}'

Create anomaly detection watcher

Set up a watcher to detect unusual network traffic patterns and potential DDoS attempts.

curl -X PUT "localhost:9200/_watcher/watch/network_anomaly_detector" -H 'Content-Type: application/json' -u elastic:your-password -d'
{
  "trigger": {
    "schedule": {
      "interval": "2m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["network-*"],
        "body": {
          "size": 0,
          "query": {
            "range": {
              "@timestamp": {
                "gte": "now-10m"
              }
            }
          },
          "aggs": {
            "requests_per_minute": {
              "date_histogram": {
                "field": "@timestamp",
                "fixed_interval": "1m"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.requests_per_minute.buckets": {
        "path": "doc_count",
        "gt": {
          "value": 1000
        }
      }
    }
  },
  "actions": {
    "log_anomaly": {
      "logging": {
        "level": "warn",
        "text": "Network anomaly detected: High traffic volume"
      }
    },
    "notify_team": {
      "email": {
        "profile": "gmail",
        "to": ["network-ops@example.com"],
        "subject": "Network Traffic Anomaly Alert",
        "body": "Unusual network traffic pattern detected. Review immediately."
      }
    }
  }
}'

Set up Kibana alerting rules

Access Kibana web interface and create alerting rules through the UI. Open your browser to configure additional alerts.

# Get the Kibana enrollment token
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Access Kibana at http://your-server-ip:5601
echo "Access Kibana at: http://$(hostname -I | awk '{print $1}'):5601"

Configure firewall rules

Open necessary ports for Kibana web interface while maintaining security.

sudo ufw allow 5601/tcp comment "Kibana web interface"
sudo ufw allow from 203.0.113.0/24 to any port 9200 comment "Elasticsearch API from trusted network"
sudo ufw reload

sudo firewall-cmd --permanent --add-port=5601/tcp
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="9200" accept'
sudo firewall-cmd --reload

Create security monitoring dashboard

Set up comprehensive security monitoring dashboards in Kibana to visualize threats and alerts in real-time. This integrates with existing ELK stack configurations for comprehensive log analysis.

curl -X POST "localhost:9200/_template/security-monitoring" -H 'Content-Type: application/json' -u elastic:your-password -d'
{
  "index_patterns": ["security-*"],
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "event_type": { "type": "keyword" },
      "severity": { "type": "keyword" },
      "source_ip": { "type": "ip" },
      "destination_ip": { "type": "ip" },
      "user_agent": { "type": "text" },
      "status_code": { "type": "integer" },
      "threat_score": { "type": "float" }
    }
  }
}'

Set up automated response actions

Configure automated responses that can block IPs or escalate alerts based on threat severity levels.

curl -X PUT "localhost:9200/_watcher/watch/automated_ip_blocker" -H 'Content-Type: application/json' -u elastic:your-password -d'
{
  "trigger": {
    "schedule": {
      "interval": "30s"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["security-*"],
        "body": {
          "query": {
            "bool": {
              "must": [
                { "range": { "threat_score": { "gte": 8.0 } } },
                { "range": { "@timestamp": { "gte": "now-2m" } } }
              ]
            }
          },
          "aggs": {
            "top_threat_ips": {
              "terms": {
                "field": "source_ip",
                "size": 5
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 1
      }
    }
  },
  "actions": {
    "block_threat_ips": {
      "webhook": {
        "scheme": "http",
        "host": "localhost",
        "port": 8080,
        "method": "post",
        "path": "/api/block-ip",
        "params": {},
        "headers": {
          "Content-Type": "application/json"
        },
        "body": "{{#ctx.payload.aggregations.top_threat_ips.buckets}}{{key}},{{/ctx.payload.aggregations.top_threat_ips.buckets}}"
      }
    },
    "escalate_alert": {
      "email": {
        "profile": "gmail",
        "to": ["security-team@example.com"],
        "subject": "HIGH PRIORITY: Automated Threat Response Triggered",
        "body": "Critical threats detected. Automated IP blocking initiated. Review security dashboard immediately."
      }
    }
  }
}'