Configure OSSEC Host-based Intrusion Detection System with nftables firewall integration for real-time threat detection and automated response. Monitor system activity, detect intrusions, and automatically block malicious IPs using active response mechanisms.

Prerequisites

Root or sudo access
Basic knowledge of Linux system administration
Understanding of firewall concepts
Email server configured (postfix)
Minimum 2GB RAM and 10GB disk space

What this solves

OSSEC HIDS provides comprehensive host-based intrusion detection by monitoring log files, system integrity, and network activity in real-time. Integrating OSSEC with nftables enables automated threat response by dynamically blocking suspicious IP addresses and implementing firewall rules based on detected security events. This setup creates a proactive security layer that responds to threats without manual intervention.

Step-by-step installation

Update system packages and install dependencies

Start by updating your package manager and installing required dependencies for OSSEC compilation and nftables management.

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential gcc make wget curl unzip
sudo apt install -y libevent-dev libssl-dev libpcre2-dev zlib1g-dev
sudo apt install -y nftables postfix mailutils

sudo dnf update -y
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y wget curl unzip libevent-devel openssl-devel
sudo dnf install -y pcre2-devel zlib-devel nftables postfix mailx

Create OSSEC user and directories

Create a dedicated user for OSSEC operations and set up the required directory structure with proper ownership.

sudo groupadd ossec
sudo useradd -d /var/ossec -s /bin/false -g ossec ossec
sudo useradd -d /var/ossec -s /bin/false -g ossec ossecm
sudo useradd -d /var/ossec -s /bin/false -g ossec ossecr

Download and compile OSSEC HIDS

Download the latest OSSEC source code and compile it with active response support enabled.

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar -xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0
sudo ./install.sh

During installation, select the following options:

Installation type: local
Email notification: yes
Email address: your-admin@example.com
SMTP server: 127.0.0.1
Run integrity check daemon: yes
Run rootkit detection engine: yes
Enable active response: yes

Configure OSSEC main settings

Configure the main OSSEC settings including email notifications, log monitoring, and active response thresholds.

<ossec_config>
  <global>
    <email_notification>yes</email_notification>
    <email_to>security@example.com</email_to>
    <smtp_server>127.0.0.1</smtp_server>
    <email_from>ossec@example.com</email_from>
    <email_maxperhour>12</email_maxperhour>
    <email_log_source>alerts.log</email_log_source>
    <agents_disconnection_time>600</agents_disconnection_time>
    <agents_disconnection_alert_time>1800</agents_disconnection_alert_time>
  </global>

  <rules>
    <include>rules_config.xml</include>
    <include>pam_rules.xml</include>
    <include>sshd_rules.xml</include>
    <include>telnetd_rules.xml</include>
    <include>syslog_rules.xml</include>
    <include>arpwatch_rules.xml</include>
    <include>symantec-av_rules.xml</include>
    <include>symantec-ws_rules.xml</include>
    <include>pix_rules.xml</include>
    <include>named_rules.xml</include>
    <include>smbd_rules.xml</include>
    <include>vsftpd_rules.xml</include>
    <include>pure-ftpd_rules.xml</include>
    <include>proftpd_rules.xml</include>
    <include>ms_ftpd_rules.xml</include>
    <include>ftpd_rules.xml</include>
    <include>hordeimp_rules.xml</include>
    <include>roundcube_rules.xml</include>
    <include>wordpress_rules.xml</include>
    <include>cimserver_rules.xml</include>
    <include>vpopmail_rules.xml</include>
    <include>vmpop3d_rules.xml</include>
    <include>courier_rules.xml</include>
    <include>web_rules.xml</include>
    <include>web_appsec_rules.xml</include>
    <include>apache_rules.xml</include>
    <include>nginx_rules.xml</include>
    <include>php_rules.xml</include>
    <include>mysql_rules.xml</include>
    <include>postgresql_rules.xml</include>
    <include>ids_rules.xml</include>
    <include>squid_rules.xml</include>
    <include>firewall_rules.xml</include>
    <include>cisco-ios_rules.xml</include>
    <include>netscreenfw_rules.xml</include>
    <include>sonicwall_rules.xml</include>
    <include>postfix_rules.xml</include>
    <include>sendmail_rules.xml</include>
    <include>imapd_rules.xml</include>
    <include>mailscanner_rules.xml</include>
    <include>dovecot_rules.xml</include>
    <include>ms-exchange_rules.xml</include>
    <include>racoon_rules.xml</include>
    <include>vpn_concentrator_rules.xml</include>
    <include>spamd_rules.xml</include>
    <include>msauth_rules.xml</include>
    <include>mcafee_av_rules.xml</include>
    <include>trend-osce_rules.xml</include>
    <include>ms-se_rules.xml</include>
    <include>zeus_rules.xml</include>
    <include>solaris_bsm_rules.xml</include>
    <include>vmware_rules.xml</include>
    <include>ms_dhcp_rules.xml</include>
    <include>asterisk_rules.xml</include>
    <include>ossec_rules.xml</include>
    <include>attack_rules.xml</include>
    <include>local_rules.xml</include>
  </rules>

  <syscheck>
    <frequency>7200</frequency>
    <alert_new_files>yes</alert_new_files>
    <auto_ignore>no</auto_ignore>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin,/boot</directories>
    <directories check_all="yes" realtime="yes">/var/www</directories>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
    <ignore>/etc/random-seed</ignore>
    <ignore>/etc/random.seed</ignore>
    <ignore>/etc/adjtime</ignore>
    <ignore>/etc/httpd/logs</ignore>
    <ignore>/etc/utmpx</ignore>
    <ignore>/etc/wtmpx</ignore>
    <ignore>/etc/cups/certs</ignore>
    <ignore>/etc/dumpdates</ignore>
    <ignore>/etc/svc/volatile</ignore>
  </syscheck>

  <rootcheck>
    <disabled>no</disabled>
    <check_files>yes</check_files>
    <check_trojans>yes</check_trojans>
    <check_dev>yes</check_dev>
    <check_sys>yes</check_sys>
    <check_pids>yes</check_pids>
    <check_ports>yes</check_ports>
    <check_if>yes</check_if>
    <frequency>7200</frequency>
    <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
    <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
  </rootcheck>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/messages</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/dpkg.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/apache2/error.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/apache2/access.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/nginx/error.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/nginx/access.log</location>
  </localfile>
</ossec_config>

Create nftables active response script

Create a custom active response script that integrates with nftables to automatically block malicious IP addresses.

#!/bin/bash
OSSEC nftables active response script
Author: OSSEC Project
License: GPL

Getting action and user
ACTION=$1
USER=$2
IP=$3

Logging
LOG="logger -t ossec-nftables"

Action function
if [ "x${ACTION}" = "xadd" ]; then
    # Check if IP is already blocked
    if ! nft list set ip filter ossec_blocked_ips 2>/dev/null | grep -q "$IP"; then
        # Add IP to nftables blocked set
        nft add element ip filter ossec_blocked_ips { $IP }
        if [ $? -eq 0 ]; then
            $LOG "OSSEC: Blocked IP $IP via nftables"
            echo "date + $IP" >> /var/ossec/logs/active-responses.log
        else
            $LOG "OSSEC: Error blocking IP $IP via nftables"
        fi
    fi
elif [ "x${ACTION}" = "xdelete" ]; then
    # Remove IP from nftables blocked set
    nft delete element ip filter ossec_blocked_ips { $IP } 2>/dev/null
    if [ $? -eq 0 ]; then
        $LOG "OSSEC: Unblocked IP $IP via nftables"
        echo "date - $IP" >> /var/ossec/logs/active-responses.log
    else
        $LOG "OSSEC: Error unblocking IP $IP via nftables (may not exist)"
    fi
else
    $LOG "OSSEC: Invalid action $ACTION for nftables-drop script"
fi

exit 0

sudo chmod 750 /var/ossec/active-response/bin/nftables-drop.sh
sudo chown root:ossec /var/ossec/active-response/bin/nftables-drop.sh

Configure nftables with OSSEC integration

Set up nftables rules with a dedicated set for OSSEC-blocked IP addresses and appropriate drop rules.

#!/usr/sbin/nft -f

Clear existing rules
flush ruleset

Define variables
define ALLOWED_SSH_IPS = { 203.0.113.0/24, 192.168.1.0/24 }
define WEB_PORTS = { 80, 443 }
define SSH_PORT = 22

Main table for filtering
table ip filter {
    # Set for OSSEC blocked IPs
    set ossec_blocked_ips {
        type ipv4_addr
        flags dynamic,timeout
        timeout 1h
        gc-interval 5m
    }

    # Set for rate limiting
    set ssh_bruteforce {
        type ipv4_addr . inet_service
        flags dynamic,timeout
        timeout 10m
    }

    # Input chain
    chain input {
        type filter hook input priority filter; policy drop;
        
        # Allow loopback traffic
        iif lo accept
        
        # Drop packets from OSSEC blocked IPs immediately
        ip saddr @ossec_blocked_ips counter drop
        
        # Allow established and related connections
        ct state established,related accept
        
        # Allow ICMP
        icmp type { destination-unreachable, router-solicitation, router-advertisement, time-exceeded, parameter-problem } accept
        icmp type echo-request limit rate 5/second accept
        
        # SSH with rate limiting and IP restriction
        tcp dport $SSH_PORT ip saddr $ALLOWED_SSH_IPS ct state new limit rate 3/minute accept
        
        # SSH brute force protection
        tcp dport $SSH_PORT add @ssh_bruteforce { ip saddr . tcp dport limit rate 3/minute burst 3 packets }
        tcp dport $SSH_PORT ip saddr . tcp dport @ssh_bruteforce counter drop
        
        # Web services
        tcp dport $WEB_PORTS accept
        
        # Drop invalid packets
        ct state invalid counter drop
        
        # Log dropped packets (sample only to avoid log flooding)
        limit rate 10/minute counter log prefix "nftables-drop: "
        
        # Default drop
        counter drop
    }

    # Forward chain
    chain forward {
        type filter hook forward priority filter; policy drop;
        
        # Drop packets from OSSEC blocked IPs
        ip saddr @ossec_blocked_ips counter drop
        ip daddr @ossec_blocked_ips counter drop
    }

    # Output chain
    chain output {
        type filter hook output priority filter; policy accept;
    }
}

Enable and configure nftables service

Enable nftables to start on boot and apply the configuration rules.

sudo systemctl enable nftables
sudo systemctl start nftables
sudo nft list ruleset

Configure OSSEC active response

Add the active response configuration to OSSEC to use the nftables script for blocking malicious IPs.

  <command>
    <name>nftables-drop</name>
    <executable>nftables-drop.sh</executable>
    <expect>srcip</expect>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <active-response>
    <disabled>no</disabled>
    <command>nftables-drop</command>
    <location>local</location>
    <rules_id>5720,5721,5722,5723,5724,5725,5726</rules_id>
    <timeout>3600</timeout>
  </active-response>

  <active-response>
    <disabled>no</disabled>
    <command>nftables-drop</command>
    <location>local</location>
    <rules_id>40111,40121,40122</rules_id>
    <timeout>1800</timeout>
  </active-response>

  <active-response>
    <disabled>no</disabled>
    <command>nftables-drop</command>
    <location>local</location>
    <level>6</level>
    <timeout>600</timeout>
  </active-response>

Configure custom OSSEC rules

Create custom rules for enhanced threat detection including web application attacks and system intrusions.

<!-- Local rules -->

<group name="local,syslog,sshd,">
  <!-- SSH brute force attempts -->
  <rule id="100001" level="10" frequency="6" timeframe="120">
    <if_matched_sid>5716</if_matched_sid>
    <description>SSH brute force attack detected (6 attempts in 120 seconds).</description>
    <group>authentication_failures,attack,</group>
  </rule>

  <!-- Multiple SSH authentication failures -->
  <rule id="100002" level="8" frequency="4" timeframe="60">
    <if_matched_sid>5716</if_matched_sid>
    <description>Multiple SSH authentication failures from same source.</description>
    <group>authentication_failures,</group>
  </rule>

  <!-- Successful login after brute force -->
  <rule id="100003" level="12">
    <if_matched_sid>5715</if_matched_sid>
    <if_fts></if_fts>
    <options>no_email_alert</options>
    <description>Successful SSH login after brute force attempts.</description>
    <group>authentication_success,attack,</group>
  </rule>
</group>

<group name="local,web,apache,">
  <!-- Web application attacks -->
  <rule id="100010" level="8">
    <if_sid>31100</if_sid>
    <url>/admin|/wp-admin|/administrator|/phpmyadmin|/xmlrpc.php</url>
    <description>Web application admin interface access attempt.</description>
    <group>web_scan,attack,</group>
  </rule>

  <!-- SQL injection attempts -->
  <rule id="100011" level="10">
    <if_sid>31100,31101,31102,31103,31104,31105,31106,31107,31108</if_sid>
    <url>union|select|insert|delete|drop|create|alter|exec|script</url>
    <description>SQL injection attempt detected.</description>
    <group>web_scan,attack,sql_injection,</group>
  </rule>

  <!-- Web shell upload attempts -->
  <rule id="100012" level="12">
    <if_sid>31100,31101,31102,31103,31104,31105,31106,31107,31108</if_sid>
    <url>.php.|.asp.|.jsp.|eval(|system(|exec(|shell_exec</url>
    <description>Web shell upload or execution attempt.</description>
    <group>web_scan,attack,webshell,</group>
  </rule>
</group>

<group name="local,system,">
  <!-- System file modifications -->
  <rule id="100020" level="8">
    <if_sid>550</if_sid>
    <field name="file">^/etc/passwd$|^/etc/shadow$|^/etc/sudoers$</field>
    <description>Critical system file modified.</description>
    <group>syscheck,critical_file,</group>
  </rule>

  <!-- Suspicious process execution -->
  <rule id="100021" level="8">
    <if_sid>2833,2834</if_sid>
    <regex>nc |netcat |telnet |wget |curl |python -c|perl -e|bash -i</regex>
    <description>Suspicious process execution detected.</description>
    <group>process,attack,</group>
  </rule>
</group>

Set proper permissions and ownership

Configure correct ownership and permissions for OSSEC files to ensure secure operation.

Never use chmod 777. It gives every user on the system full access to your files. Instead, use minimal permissions and correct ownership for security.

sudo chown -R root:ossec /var/ossec
sudo chmod -R 750 /var/ossec
sudo chmod 640 /var/ossec/etc/ossec.conf
sudo chmod 750 /var/ossec/bin/*
sudo chmod 755 /var/ossec/logs
sudo chmod 644 /var/ossec/logs/*.log 2>/dev/null || true
sudo touch /var/ossec/logs/active-responses.log
sudo chown ossec:ossec /var/ossec/logs/active-responses.log
sudo chmod 644 /var/ossec/logs/active-responses.log

Start and enable OSSEC service

Start the OSSEC service and enable it to start automatically on boot.

sudo systemctl enable ossec
sudo systemctl start ossec
sudo systemctl status ossec

Configure email notifications

Set up email notifications for OSSEC alerts using postfix for immediate threat notifications.

sudo dpkg-reconfigure postfix

sudo systemctl enable postfix
sudo systemctl start postfix

Configure postfix as "Internet Site" and set your domain name when prompted.

Verify your setup

Test the OSSEC installation and nftables integration to ensure proper functionality.

# Check OSSEC status
sudo /var/ossec/bin/ossec-control status

Verify nftables rules
sudo nft list ruleset

Check OSSEC blocked IPs set
sudo nft list set ip filter ossec_blocked_ips

Test email functionality
echo "OSSEC test email" | mail -s "OSSEC Test" security@example.com

Check OSSEC logs
sudo tail -f /var/ossec/logs/ossec.log

Monitor active responses
sudo tail -f /var/ossec/logs/active-responses.log

Test SSH brute force detection (from another machine)
ssh -o ConnectTimeout=1 -o PreferredAuthentications=password user@your-server-ip

Check system integrity
sudo /var/ossec/bin/ossec-control restart
sudo /var/ossec/bin/syscheck_control -i

Verify rule loading
sudo /var/ossec/bin/ossec-logtest

Common issues

Symptom	Cause	Fix
OSSEC won't start	Incorrect permissions or missing dependencies	Check `sudo /var/ossec/bin/ossec-control status` and verify file ownership with `chown -R root:ossec /var/ossec`
Active response not working	Script permissions or nftables not configured	Verify script is executable: `chmod 750 /var/ossec/active-response/bin/nftables-drop.sh`
No email alerts received	Postfix not configured or firewall blocking SMTP	Test postfix: `echo "test" \| mail -s "test" user@example.com` and check `/var/log/mail.log`
nftables rules not persisting	nftables service not enabled	Enable service: `systemctl enable nftables`
High false positive rate	Overly sensitive rules	Adjust rule levels in `/var/ossec/rules/local_rules.xml` and restart OSSEC
IPs not getting unblocked	Timeout not working correctly	Check nftables set timeout: `nft list set ip filter ossec_blocked_ips`

Next steps

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash
set -euo pipefail

# OSSEC HIDS with nftables Integration Installation Script
# Supports Ubuntu, Debian, AlmaLinux, Rocky Linux, CentOS, RHEL, Fedora

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Default values
ADMIN_EMAIL="${1:-admin@localhost}"
SMTP_SERVER="${2:-127.0.0.1}"
OSSEC_VERSION="3.7.0"

usage() {
    echo "Usage: $0 [admin_email] [smtp_server]"
    echo "Example: $0 security@example.com 127.0.0.1"
    exit 1
}

log() {
    echo -e "${GREEN}[INFO]${NC} $1"
}

warn() {
    echo -e "${YELLOW}[WARN]${NC} $1"
}

error() {
    echo -e "${RED}[ERROR]${NC} $1" >&2
    exit 1
}

cleanup() {
    warn "Installation failed. Cleaning up..."
    [ -d "/tmp/ossec-hids-${OSSEC_VERSION}" ] && rm -rf "/tmp/ossec-hids-${OSSEC_VERSION}"
    [ -f "/tmp/${OSSEC_VERSION}.tar.gz" ] && rm -f "/tmp/${OSSEC_VERSION}.tar.gz"
}

trap cleanup ERR

# Check prerequisites
echo "[1/10] Checking prerequisites..."
if [[ $EUID -ne 0 ]]; then
    error "This script must be run as root"
fi

# Validate email format
if [[ ! $ADMIN_EMAIL =~ ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$ ]]; then
    error "Invalid email format: $ADMIN_EMAIL"
fi

# Detect distribution
echo "[2/10] Detecting distribution..."
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian)
            PKG_MGR="apt"
            PKG_UPDATE="apt update && apt upgrade -y"
            PKG_INSTALL="apt install -y"
            DEV_TOOLS="build-essential gcc make"
            DEPS="libevent-dev libssl-dev libpcre2-dev zlib1g-dev"
            MAIL_DEPS="postfix mailutils"
            ;;
        almalinux|rocky|centos|rhel|ol)
            PKG_MGR="dnf"
            PKG_UPDATE="dnf update -y"
            PKG_INSTALL="dnf install -y"
            DEV_TOOLS="gcc gcc-c++ make"
            DEPS="libevent-devel openssl-devel pcre2-devel zlib-devel"
            MAIL_DEPS="postfix mailx"
            # Install Development Tools group
            dnf groupinstall -y "Development Tools" || true
            ;;
        fedora)
            PKG_MGR="dnf"
            PKG_UPDATE="dnf update -y"
            PKG_INSTALL="dnf install -y"
            DEV_TOOLS="gcc gcc-c++ make"
            DEPS="libevent-devel openssl-devel pcre2-devel zlib-devel"
            MAIL_DEPS="postfix mailx"
            dnf groupinstall -y "Development Tools" || true
            ;;
        amzn)
            PKG_MGR="yum"
            PKG_UPDATE="yum update -y"
            PKG_INSTALL="yum install -y"
            DEV_TOOLS="gcc gcc-c++ make"
            DEPS="libevent-devel openssl-devel pcre2-devel zlib-devel"
            MAIL_DEPS="postfix mailx"
            yum groupinstall -y "Development Tools" || true
            ;;
        *)
            error "Unsupported distribution: $ID"
            ;;
    esac
    log "Detected: $PRETTY_NAME"
else
    error "Cannot detect distribution. /etc/os-release not found."
fi

# Update system packages
echo "[3/10] Updating system packages..."
eval $PKG_UPDATE

# Install dependencies
echo "[4/10] Installing dependencies..."
$PKG_INSTALL wget curl unzip nftables $DEV_TOOLS $DEPS $MAIL_DEPS

# Create OSSEC users
echo "[5/10] Creating OSSEC users and groups..."
if ! getent group ossec > /dev/null; then
    groupadd ossec
fi

for user in ossec ossecm ossecr; do
    if ! id "$user" > /dev/null 2>&1; then
        useradd -d /var/ossec -s /bin/false -g ossec "$user"
    fi
done

# Download OSSEC
echo "[6/10] Downloading OSSEC HIDS..."
cd /tmp
if [ ! -f "${OSSEC_VERSION}.tar.gz" ]; then
    wget "https://github.com/ossec/ossec-hids/archive/${OSSEC_VERSION}.tar.gz"
fi

if [ -d "ossec-hids-${OSSEC_VERSION}" ]; then
    rm -rf "ossec-hids-${OSSEC_VERSION}"
fi

tar -xzf "${OSSEC_VERSION}.tar.gz"
cd "ossec-hids-${OSSEC_VERSION}"

# Install OSSEC with preseeded answers
echo "[7/10] Installing OSSEC HIDS..."
cat > /tmp/ossec_install_answers << EOF
en
local
$ADMIN_EMAIL
y
$SMTP_SERVER
y
y
y
EOF

./install.sh < /tmp/ossec_install_answers

# Configure OSSEC
echo "[8/10] Configuring OSSEC..."
cat > /var/ossec/etc/ossec.conf << EOF
<ossec_config>
  <global>
    <email_notification>yes</email_notification>
    <email_to>$ADMIN_EMAIL</email_to>
    <smtp_server>$SMTP_SERVER</smtp_server>
    <email_from>ossec@$(hostname)</email_from>
    <email_maxperhour>12</email_maxperhour>
    <email_log_source>alerts.log</email_log_source>
    <agents_disconnection_time>600</agents_disconnection_time>
    <agents_disconnection_alert_time>1800</agents_disconnection_alert_time>
  </global>

  <rules>
    <include>rules_config.xml</include>
    <include>pam_rules.xml</include>
    <include>sshd_rules.xml</include>
    <include>syslog_rules.xml</include>
    <include>web_rules.xml</include>
    <include>local_rules.xml</include>
  </rules>

  <syscheck>
    <frequency>79200</frequency>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin</directories>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
    <ignore>/etc/random-seed</ignore>
  </syscheck>

  <rootcheck>
    <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
    <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
  </rootcheck>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>command</log_format>
    <command>df -P</command>
    <frequency>360</frequency>
  </localfile>

  <localfile>
    <log_format>full_command</log_format>
    <command>netstat -tan |grep LISTEN |egrep -v '(127.0.0.1|::1)' | sort</command>
    <frequency>360</frequency>
  </localfile>

  <localfile>
    <log_format>full_command</log_format>
    <command>last -n 5</command>
    <frequency>360</frequency>
  </localfile>

  <active-response>
    <disabled>no</disabled>
    <command>host-deny</command>
    <location>local</location>
    <level>6</level>
    <timeout>600</timeout>
  </active-response>

  <active-response>
    <disabled>no</disabled>
    <command>nftables-drop</command>
    <location>local</location>
    <level>6</level>
    <timeout>600</timeout>
  </active-response>
</ossec_config>
EOF

# Create nftables active response script
echo "[9/10] Setting up nftables integration..."
cat > /var/ossec/active-response/bin/nftables-drop.sh << 'EOF'
#!/bin/bash
# nftables active response script for OSSEC

ACTION=$1
USER=$2
IP=$3

if [ "x$IP" = "x" ]; then
    exit 1
fi

if [ "$ACTION" = "add" ]; then
    nft add rule ip filter input ip saddr $IP counter drop
elif [ "$ACTION" = "delete" ]; then
    nft delete rule ip filter input ip saddr $IP counter drop 2>/dev/null || true
fi

exit 0
EOF

chmod 750 /var/ossec/active-response/bin/nftables-drop.sh
chown root:ossec /var/ossec/active-response/bin/nftables-drop.sh

# Add nftables command to OSSEC
cat >> /var/ossec/etc/ossec.conf << 'EOF'
  <command>
    <name>nftables-drop</name>
    <executable>nftables-drop.sh</executable>
    <expect>srcip</expect>
    <timeout_allowed>yes</timeout_allowed>
  </command>
EOF

# Set proper ownership and permissions
chown -R root:ossec /var/ossec/etc/
chmod -R 640 /var/ossec/etc/
chmod 750 /var/ossec/etc/

# Enable and start services
systemctl enable nftables
systemctl start nftables
systemctl enable postfix
systemctl start postfix

# Start OSSEC
/var/ossec/bin/ossec-control start

# Verification
echo "[10/10] Verifying installation..."
sleep 5

if pgrep -f ossec-analysisd > /dev/null; then
    log "✓ OSSEC analysis daemon is running"
else
    error "✗ OSSEC analysis daemon failed to start"
fi

if systemctl is-active nftables > /dev/null; then
    log "✓ nftables service is active"
else
    error "✗ nftables service is not active"
fi

if [ -f /var/ossec/logs/ossec.log ]; then
    log "✓ OSSEC log file created"
else
    warn "OSSEC log file not found yet"
fi

# Cleanup
rm -f /tmp/ossec_install_answers
rm -rf "/tmp/ossec-hids-${OSSEC_VERSION}"
rm -f "/tmp/${OSSEC_VERSION}.tar.gz"

log "OSSEC HIDS installation completed successfully!"
log "Admin email: $ADMIN_EMAIL"
log "SMTP server: $SMTP_SERVER"
log "Configuration file: /var/ossec/etc/ossec.conf"
log "Log files: /var/ossec/logs/"
log "Control script: /var/ossec/bin/ossec-control"
warn "Remember to configure your mail server for email notifications"

Review the script before running. Execute with: bash install.sh

#ossec #hids #nftables #intrusion-detection #active-response

Set up intrusion detection with OSSEC HIDS and nftables integration for automated threat response

Prerequisites

What this solves

Step-by-step installation

Update system packages and install dependencies

Create OSSEC user and directories

Download and compile OSSEC HIDS

Configure OSSEC main settings

Create nftables active response script

OSSEC nftables active response script

Author: OSSEC Project

License: GPL

Getting action and user

Logging

Action function

Configure nftables with OSSEC integration

Clear existing rules

Define variables

Main table for filtering

Enable and configure nftables service

Configure OSSEC active response

Configure custom OSSEC rules

Set proper permissions and ownership

Start and enable OSSEC service

Configure email notifications

Verify your setup

Verify nftables rules

Check OSSEC blocked IPs set

Test email functionality

Check OSSEC logs

Monitor active responses

Test SSH brute force detection (from another machine)

ssh -o ConnectTimeout=1 -o PreferredAuthentications=password user@your-server-ip

Check system integrity

Verify rule loading

Common issues

Next steps

Related tutorials

Set up NGINX web application firewall with ModSecurity 3 and OWASP Core Rule Set

Implement network security with iptables connection tracking and stateful packet filtering

Configure ClickHouse users and RBAC for production environments with authentication and access control

Don't want to manage this yourself?