Configure Linux audit system with SIEM integration for Splunk security monitoring

Install audit framework

Install auditd and audit utilities for comprehensive system monitoring. The audit framework provides kernel-level event capture and user-space processing tools.

sudo apt update
sudo apt install -y auditd audispd-plugins audit-utils

sudo dnf install -y audit audit-libs audispd-plugins

Configure audit rules for security monitoring

Create comprehensive audit rules covering file access, system calls, privileged operations, and user activities. These rules capture security-relevant events for SIEM analysis.

# Delete existing rules and set buffer size
-D
-b 8192
-f 1

Monitor authentication events
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k identity
-w /etc/ssh/sshd_config -p wa -k sshd

Monitor system configuration changes
-w /etc/hosts -p wa -k network
-w /etc/resolv.conf -p wa -k network
-w /etc/crontab -p wa -k cron
-w /etc/cron.allow -p wa -k cron
-w /etc/cron.deny -p wa -k cron
-w /var/spool/cron/ -p wa -k cron

Monitor privileged commands
-w /usr/bin/sudo -p x -k privileged
-w /usr/bin/su -p x -k privileged
-w /bin/mount -p x -k privileged
-w /bin/umount -p x -k privileged
-w /usr/bin/passwd -p x -k privileged

Monitor network configuration
-a always,exit -F arch=b64 -S socket,connect,accept,bind,listen -k network
-a always,exit -F arch=b32 -S socket,connect,accept,bind,listen -k network

Monitor file system operations
-a always,exit -F arch=b64 -S unlink,unlinkat,rename,renameat -F success=1 -k delete
-a always,exit -F arch=b32 -S unlink,unlinkat,rename,renameat -F success=1 -k delete

Monitor process execution
-a always,exit -F arch=b64 -S execve -k execution
-a always,exit -F arch=b32 -S execve -k execution

Monitor system calls for privilege escalation
-a always,exit -F arch=b64 -S setuid,setgid,setreuid,setregid -F success=1 -k privilege_escalation
-a always,exit -F arch=b32 -S setuid,setgid,setreuid,setregid -F success=1 -k privilege_escalation

Make configuration immutable
-e 2

Configure auditd daemon settings

Optimize auditd performance and configure log rotation, buffering, and failure handling for production environments.

# Log file configuration
log_file = /var/log/audit/audit.log
log_format = ENRICHED
log_group = adm
priority_boost = 4
flush = INCREMENTAL_ASYNC
freq = 50
num_logs = 5
disp_qos = lossy
dispatcher = /sbin/audispd
name_format = HOSTNAME
max_log_file = 100
max_log_file_action = ROTATE
space_left = 75
space_left_action = SYSLOG
verify_email = yes
action_mail_acct = root
admin_space_left = 50
admin_space_left_action = SUSPEND
disk_full_action = SUSPEND
disk_error_action = SUSPEND
use_libwrap = yes
tcp_listen_queue = 5
tcp_max_per_addr = 1
tcp_client_max_idle = 0
enable_krb5 = no
krb5_principal = auditd

Install Splunk Universal Forwarder

Download and install Splunk Universal Forwarder for shipping audit logs to your Splunk indexer. This lightweight agent handles log forwarding and parsing.

cd /tmp
wget -O splunkforwarder-9.1.2-b6b9c8185839-linux-2.6-x86_64.rpm 'https://download.splunk.com/products/universalforwarder/releases/9.1.2/linux/splunkforwarder-9.1.2-b6b9c8185839-linux-2.6-x86_64.rpm'
sudo groupadd splunk
sudo useradd -r -g splunk -d /opt/splunkforwarder -s /bin/bash splunk

sudo dpkg --install /tmp/splunkforwarder-9.1.2-b6b9c8185839-linux-2.6-x86_64.rpm

sudo rpm -ivh /tmp/splunkforwarder-9.1.2-b6b9c8185839-linux-2.6-x86_64.rpm

Configure Splunk Universal Forwarder

Set up the forwarder to monitor audit logs and configure connection to your Splunk indexer. Replace the indexer address with your actual Splunk server.

sudo chown -R splunk:splunk /opt/splunkforwarder
sudo -u splunk /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt --seed-passwd changeme123
sudo -u splunk /opt/splunkforwarder/bin/splunk stop

[tcpout]
defaultGroup = splunk_indexers

[tcpout:splunk_indexers]
server = 203.0.113.10:9997
compressed = true
useACK = true

[tcpout-server://203.0.113.10:9997]
Replace with your Splunk indexer IP

Configure audit log monitoring

Set up inputs configuration to monitor audit logs with proper source typing and parsing for Splunk ingestion.

[monitor:///var/log/audit/audit.log]
disabled = false
index = security
sourcetype = linux:audit
host_segment = 4

[monitor:///var/log/auth.log]
disabled = false
index = security
sourcetype = linux:auth
host_segment = 4

[monitor:///var/log/secure]
disabled = false
index = security
sourcetype = linux:secure
host_segment = 4

[monitor:///var/log/syslog]
disabled = false
index = security
sourcetype = linux:syslog
host_segment = 4

Configure audit log parsing

Create field extraction and parsing rules for audit events to enable structured searching and analysis in Splunk.

[linux:audit]
KV_MODE = auto
SHOULD_LINEMERGE = false
LINE_BREAKER = (\r?\n)type=
TRUNCATEDE = 10000
TIME_PREFIX = msg=audit\(
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_FORMAT = %s.%3N:%f
REPORT-audit = audit_kv
EVAL-vendor_product = "Linux auditd"

[linux:auth]
SHOULD_LINEMERGE = false
TRUNCATE = 10000
KV_MODE = auto
REPORT-auth = auth_extractions

[linux:secure]
SHOULD_LINEMERGE = false
TRUNCATE = 10000
KV_MODE = auto
REPORT-secure = secure_extractions

Create field extraction rules

Define regex patterns to extract key fields from audit logs for structured analysis and alerting.

[audit_kv]
REGEX = (\w+)=([^\s]+)
FORMAT = $1::$2
MV_ADD = true

[auth_extractions]
REGEX = (\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})\s+(\S+)\s+(\w+)(?:\[\d+\])?:\s*(.+)
FORMAT = timestamp::$1 host::$2 program::$3 message::$4

[secure_extractions]
REGEX = (\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})\s+(\S+)\s+(\w+)(?:\[\d+\])?:\s*(.+)
FORMAT = timestamp::$1 host::$2 program::$3 message::$4

Set up file permissions and ownership

Configure proper ownership and permissions for Splunk to access audit logs. Add the splunk user to the adm group for log file access.

sudo usermod -a -G adm splunk
sudo chmod 640 /var/log/audit/audit.log
sudo chown root:adm /var/log/audit/audit.log
sudo chown -R splunk:splunk /opt/splunkforwarder

Enable and start services

Start auditd and Splunk Universal Forwarder, then enable them to start on system boot. Verify both services are running correctly.

sudo systemctl enable auditd
sudo systemctl start auditd
sudo systemctl status auditd

sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
sudo systemctl start SplunkForwarder
sudo systemctl status SplunkForwarder

Configure real-time alerting

Set up audit event processing for real-time analysis and alerting. This configuration sends high-priority events immediately to Splunk.

active = yes
direction = out
path = builtin_syslog
type = builtin
args = LOG_INFO
format = string

priority = LOG_INFO
facility = LOG_LOCAL0

Generate test audit events

Create test events to verify audit logging and Splunk forwarding are working correctly. These commands will generate various audit events for validation.

sudo touch /tmp/audit_test_file
sudo chmod 755 /tmp/audit_test_file
sudo rm /tmp/audit_test_file
sudo su - root -c 'whoami'
sudo useradd -r testaudituser
sudo userdel testaudituser

Set up audit event correlation

Configure audit rules for advanced threat detection and correlation. These rules help identify suspicious activity patterns.

# Monitor for suspicious shell activity
-w /bin/bash -p x -k suspicious_shell
-w /bin/sh -p x -k suspicious_shell
-w /usr/bin/nc -p x -k suspicious_network
-w /usr/bin/netcat -p x -k suspicious_network

Monitor for data exfiltration
-w /usr/bin/wget -p x -k data_exfiltration
-w /usr/bin/curl -p x -k data_exfiltration
-w /usr/bin/scp -p x -k data_exfiltration
-w /usr/bin/rsync -p x -k data_exfiltration

Monitor kernel module operations
-w /sbin/insmod -p x -k kernel_modules
-w /sbin/rmmod -p x -k kernel_modules
-w /sbin/modprobe -p x -k kernel_modules

Monitor system time changes
-a always,exit -F arch=b64 -S adjtimex,settimeofday,stime -k time_change
-a always,exit -F arch=b32 -S adjtimex,settimeofday,stime -k time_change
-a always,exit -F arch=b64 -S clock_settime -k time_change
-a always,exit -F arch=b32 -S clock_settime -k time_change

Monitor failed login attempts
-w /var/log/faillog -p wa -k logins
-w /var/log/lastlog -p wa -k logins

Configure Splunk searches and alerts

Create saved searches in Splunk for common security scenarios. These searches help detect and alert on suspicious activities.

# Failed login attempts
index=security sourcetype=linux:auth "Failed password" | stats count by src_ip user | where count > 5

Privilege escalation attempts
index=security sourcetype=linux:audit key=privilege_escalation | stats count by exe user | sort -count

Suspicious file deletions
index=security sourcetype=linux:audit key=delete | stats count by exe user | sort -count

Network connections from unusual processes
index=security sourcetype=linux:audit key=network | stats count by exe user | sort -count

Kernel module installations
index=security sourcetype=linux:audit key=kernel_modules | table _time user exe

Time manipulation attempts
index=security sourcetype=linux:audit key=time_change | table _time user exe syscall