Set up Fluentd with td-agent for enterprise-grade log collection, custom parsers, and multi-destination routing. Configure SSL, performance optimization, and security hardening for production environments.

Prerequisites

Root or sudo access
At least 2GB RAM
Network access to log destinations
Basic understanding of system logs

What this solves

Fluentd is a unified logging layer that collects, transforms, and routes log data from multiple sources to various destinations. This tutorial shows you how to install td-agent (the stable Fluentd distribution), configure input plugins for system and application logs, set up custom parsers for different log formats, and optimize performance for production environments.

Step-by-step installation

Update system packages

Start by updating your package manager to ensure you get the latest versions of dependencies.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install required dependencies

Install curl and gnupg for repository setup and SSL certificate verification.

sudo apt install -y curl gnupg lsb-release ca-certificates

sudo dnf install -y curl gnupg2 ca-certificates

Add Treasure Data repository

Add the official td-agent repository to get the latest stable version of Fluentd with enterprise features.

curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-td-agent4.sh | sh

curl -fsSL https://toolbelt.treasuredata.com/sh/install-redhat-td-agent4.sh | sh

Create td-agent user and directories

Set up proper ownership and permissions for log collection and processing. The td-agent user needs access to system logs and write access to its working directories.

sudo mkdir -p /var/log/td-agent /etc/td-agent/conf.d
sudo chown -R td-agent:td-agent /var/log/td-agent
sudo chmod 755 /var/log/td-agent
sudo usermod -a -G adm td-agent

Note: Adding td-agent to the adm group allows it to read system logs in /var/log without requiring overly broad permissions.

Configure main td-agent configuration

Create the primary configuration file that defines input sources, parsing rules, and output destinations.

# System configuration

  log_level info
  suppress_repeated_stacktrace true
  emit_error_log_interval 30s
  suppress_config_dump
  without_source true


Include additional configurations
@include conf.d/*.conf

Input: System logs

  @type tail
  @id syslog_input
  tag system.syslog
  path /var/log/syslog
  pos_file /var/log/td-agent/syslog.log.pos
  
    @type syslog
    message_format rfc3164
    with_priority true
  


Input: Authentication logs

  @type tail
  @id auth_input
  tag system.auth
  path /var/log/auth.log
  pos_file /var/log/td-agent/auth.log.pos
  
    @type syslog
    message_format rfc3164
    with_priority true
  


Input: Application logs with JSON format

  @type tail
  @id app_json_input
  tag app.json
  path /var/log/app/*.json
  pos_file /var/log/td-agent/app.json.pos
  
    @type json
    time_key timestamp
    time_format %Y-%m-%dT%H:%M:%S.%L%z
  


Input: NGINX access logs

  @type tail
  @id nginx_access_input
  tag web.nginx.access
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx.access.pos
  
    @type nginx
    format combined
  


Filter: Add hostname and environment tags

  @type record_transformer
  
    hostname ${hostname}
    environment production
    log_source fluentd
  


Output: File for debugging

  @type file
  path /var/log/td-agent/debug
  append true
  time_slice_format %Y%m%d
  time_slice_wait 10m
  time_format %Y%m%dT%H%M%S%z
  buffer_type file
  buffer_path /var/log/td-agent/buffer/debug
  flush_interval 30s


Output: Forward to centralized log server
 app. web.**>
  @type forward
  @id main_forward
  
    name log-server
    host 203.0.113.10
    port 24224
    weight 60
  
  
    @type file
    path /var/log/td-agent/buffer/forward
    flush_mode interval
    flush_interval 5s
    flush_thread_count 2
    retry_type exponential_backoff
    retry_wait 1s
    retry_max_interval 60s
    retry_timeout 60m
    overflow_action block

Create custom parser for application logs

Define a custom parser configuration for application-specific log formats that don't match standard patterns.

# Custom parser for application logs with specific format

  @type tail
  @id custom_app_input
  tag app.custom
  path /var/log/myapp/*.log
  pos_file /var/log/td-agent/myapp.log.pos
  
    @type regexp
    expression /^(?\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) \[(?\w+)\] (?\w+): (?.*)/
    time_format %Y-%m-%d %H:%M:%S
  


Multi-line parser for Java stack traces

  @type tail
  @id java_app_input
  tag app.java
  path /var/log/java-app/*.log
  pos_file /var/log/td-agent/java-app.log.pos
  multiline_flush_interval 5s
  
    @type multiline
    format_firstline /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/
    format1 /^(?\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) \[(?\w+)\] (?[^:]+): (?.*)/
    time_format %Y-%m-%d %H:%M:%S
  


CSV log parser

  @type tail
  @id csv_app_input
  tag app.csv
  path /var/log/csv-app/*.csv
  pos_file /var/log/td-agent/csv-app.log.pos
  
    @type csv
    keys time,level,user_id,action,result,duration
    time_key time
    time_format %Y-%m-%d %H:%M:%S

Configure multiple output destinations

Set up routing to send different types of logs to appropriate destinations with error handling and buffering.

# Output to Elasticsearch for search and analysis
 web.>
  @type elasticsearch
  @id elasticsearch_output
  host 203.0.113.20
  port 9200
  scheme https
  ssl_verify true
  user elastic
  password your-secure-password
  index_name fluentd-logs
  type_name _doc
  logstash_format true
  logstash_prefix fluentd
  logstash_dateformat %Y.%m.%d
  include_tag_key true
  tag_key @log_name
  
    @type file
    path /var/log/td-agent/buffer/elasticsearch
    flush_mode interval
    flush_interval 10s
    flush_thread_count 4
    retry_type exponential_backoff
    retry_wait 2s
    retry_max_interval 120s
    retry_timeout 120m
    overflow_action block
  


Output critical errors to email alert

  @type copy
  
    @type elasticsearch
    @id auth_elasticsearch
    host 203.0.113.20
    port 9200
    scheme https
    ssl_verify true
    user elastic
    password your-secure-password
    index_name security-logs
    type_name _doc
    logstash_format true
    logstash_prefix security
  
  
    @type grep
    
      key message
      pattern /Failed password|authentication failure|sudo/
    
    @type file
    path /var/log/td-agent/security-alerts
    append true
    time_slice_format %Y%m%d
    time_format %Y%m%dT%H%M%S%z
  


Output application logs to S3 for long-term storage

  @type s3
  @id s3_output
  aws_key_id YOUR_AWS_ACCESS_KEY
  aws_sec_key YOUR_AWS_SECRET_KEY
  s3_bucket my-log-bucket
  s3_region us-east-1
  path logs/
  s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
  time_slice_format %Y/%m/%d/%H
  store_as gzip_command
  
    @type file
    path /var/log/td-agent/buffer/s3
    timekey 3600
    timekey_wait 10m
    chunk_limit_size 256m
    flush_mode interval
    flush_interval 60s

Configure SSL and security settings

Set up SSL certificates and secure communication between Fluentd instances and output destinations.

# SSL input for receiving logs from other Fluentd instances

  @type forward
  @id secure_forward_input
  port 24224
  bind 0.0.0.0
  
    cert_path /etc/td-agent/ssl/fluentd.crt
    private_key_path /etc/td-agent/ssl/fluentd.key
    ca_path /etc/td-agent/ssl/ca.crt
    client_cert_auth true
  
  
    self_hostname fluentd-server
    shared_key secure-shared-key-change-this
  


Secure forward output with authentication

  @type secure_forward
  @id secure_forward_output
  self_hostname fluentd-client
  shared_key secure-shared-key-change-this
  secure yes
  enable_strict_verification yes
  ca_cert_path /etc/td-agent/ssl/ca.crt
  
    host 203.0.113.30
    port 24284
  
  
    @type file
    path /var/log/td-agent/buffer/secure
    flush_mode interval
    flush_interval 10s

Create SSL certificates directory

Set up the SSL directory with proper permissions for certificate storage. Only the td-agent user should have read access to private keys.

sudo mkdir -p /etc/td-agent/ssl
sudo chown td-agent:td-agent /etc/td-agent/ssl
sudo chmod 700 /etc/td-agent/ssl

Never use chmod 777. Private keys must only be readable by the service user. Use chmod 600 for private keys and 644 for certificates.

Configure performance optimization

Optimize buffer settings, memory usage, and worker processes for high-throughput log processing.

# Performance tuning for high-volume logs

  workers 4
  root_dir /var/log/td-agent
  suppress_repeated_stacktrace true
  emit_error_log_interval 30s
  suppress_config_dump true
  log_event_verbose false


Buffer optimization for high throughput

  @type forward
  @id high_volume_output
  
    name primary-server
    host 203.0.113.40
    port 24224
  
  
    @type file
    path /var/log/td-agent/buffer/high-volume
    flush_mode immediate
    flush_thread_count 8
    chunk_limit_size 64m
    chunk_limit_records 100000
    total_limit_size 2g
    overflow_action drop_oldest_chunk
    retry_type exponential_backoff
    retry_wait 1s
    retry_max_interval 60s
    retry_timeout 60m
    compress gzip

Set up log rotation and maintenance

Configure logrotate to manage Fluentd's own log files and prevent disk space issues.

/var/log/td-agent/td-agent.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    copytruncate
    su td-agent td-agent
}

/var/log/td-agent/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    copytruncate
    su td-agent td-agent
}

Enable and start td-agent service

Start the td-agent service and enable it to start automatically on system boot.

sudo systemctl enable td-agent
sudo systemctl start td-agent
sudo systemctl status td-agent

Configure firewall rules

Open necessary ports for log collection while maintaining security. Only allow connections from trusted sources.

sudo ufw allow from 203.0.113.0/24 to any port 24224
sudo ufw allow from 203.0.113.0/24 to any port 24284
sudo ufw reload

sudo firewall-cmd --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="24224" accept' --permanent
sudo firewall-cmd --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="24284" accept' --permanent
sudo firewall-cmd --reload

Verify your setup

Test your Fluentd installation and configuration with these verification commands.

# Check td-agent service status
sudo systemctl status td-agent

Verify configuration syntax
sudo td-agent --dry-run

Check td-agent version and plugins
td-agent --version
td-agent-gem list

Test log processing
echo '{"message":"test log entry","level":"info"}' | sudo -u td-agent td-agent -c /etc/td-agent/td-agent.conf --dry-run

Monitor real-time logs
sudo tail -f /var/log/td-agent/td-agent.log

Check buffer status
sudo ls -la /var/log/td-agent/buffer/

Test network connectivity to outputs
sudo netstat -tlnp | grep td-agent
telnet 203.0.113.10 24224

Performance tuning and monitoring

Monitor Fluentd performance and adjust settings based on your log volume and system resources.

Install monitoring plugins

Add plugins for monitoring Fluentd's internal metrics and performance.

sudo td-agent-gem install fluent-plugin-prometheus
sudo td-agent-gem install fluent-plugin-monitor-agent

Configure monitoring endpoints

Set up HTTP endpoints to monitor Fluentd's internal status and metrics.

# Monitor agent for internal metrics

  @type monitor_agent
  @id monitor_agent_input
  bind 127.0.0.1
  port 24220
  tag fluentd.monitor.metrics


Debug agent for configuration inspection

  @type debug_agent
  @id debug_agent_input
  bind 127.0.0.1
  port 24230


Prometheus metrics endpoint

  @type prometheus
  @id prometheus_metrics
  bind 127.0.0.1
  port 24231
  metrics_path /metrics


Internal metrics collection

  @type prometheus_monitor
  @id prometheus_monitor
  interval 30s

Common issues

Symptom	Cause	Fix
Permission denied reading logs	td-agent user lacks read access	`sudo usermod -a -G adm td-agent`
Buffer files filling disk	Output destination unreachable	Check network connectivity and increase buffer limits
High memory usage	Large buffer chunk sizes	Reduce chunk_limit_size and increase flush frequency
SSL handshake failures	Certificate mismatch or expired	Verify certificates with `openssl x509 -text -noout -in cert.crt`
Config syntax errors	Invalid YAML or plugin parameters	Run `sudo td-agent --dry-run` to validate
Logs not being forwarded	Firewall blocking connections	Check firewall rules and test with `telnet host port`
Service fails to start	Port already in use	Check with `sudo netstat -tlnp \| grep :24224`

Next steps

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash
set -euo pipefail

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Global variables
LOG_SERVER_HOST="${1:-203.0.113.10}"
ENVIRONMENT="${2:-production}"

# Usage message
usage() {
    echo "Usage: $0 [LOG_SERVER_HOST] [ENVIRONMENT]"
    echo "  LOG_SERVER_HOST: IP address of centralized log server (default: 203.0.113.10)"
    echo "  ENVIRONMENT: Environment tag for logs (default: production)"
    exit 1
}

# Cleanup function for rollback
cleanup() {
    echo -e "${RED}[ERROR] Installation failed. Cleaning up...${NC}"
    systemctl stop td-agent 2>/dev/null || true
    systemctl disable td-agent 2>/dev/null || true
    if [ "$PKG_MGR" = "apt" ]; then
        apt-get remove -y td-agent 2>/dev/null || true
    else
        $PKG_INSTALL remove -y td-agent 2>/dev/null || true
    fi
    rm -rf /etc/td-agent /var/log/td-agent
    exit 1
}

# Set error trap
trap cleanup ERR

# Check if running as root
if [[ $EUID -ne 0 ]]; then
   echo -e "${RED}This script must be run as root${NC}"
   exit 1
fi

# Detect distribution
echo -e "${YELLOW}[1/8] Detecting distribution...${NC}"
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian) 
            PKG_MGR="apt"
            PKG_INSTALL="apt install -y"
            UPDATE_CMD="apt update && apt upgrade -y"
            DEPS="curl gnupg lsb-release ca-certificates"
            INSTALL_SCRIPT="install-ubuntu-jammy-td-agent4.sh"
            ;;
        almalinux|rocky|centos|rhel|ol|fedora) 
            PKG_MGR="dnf"
            PKG_INSTALL="dnf install -y"
            UPDATE_CMD="dnf update -y"
            DEPS="curl gnupg2 ca-certificates"
            INSTALL_SCRIPT="install-redhat-td-agent4.sh"
            ;;
        amzn) 
            PKG_MGR="yum"
            PKG_INSTALL="yum install -y"
            UPDATE_CMD="yum update -y"
            DEPS="curl gnupg2 ca-certificates"
            INSTALL_SCRIPT="install-redhat-td-agent4.sh"
            ;;
        *) 
            echo -e "${RED}Unsupported distribution: $ID${NC}"
            exit 1
            ;;
    esac
    echo -e "${GREEN}Detected: $PRETTY_NAME${NC}"
else
    echo -e "${RED}Cannot detect distribution${NC}"
    exit 1
fi

# Update system packages
echo -e "${YELLOW}[2/8] Updating system packages...${NC}"
eval $UPDATE_CMD

# Install dependencies
echo -e "${YELLOW}[3/8] Installing dependencies...${NC}"
$PKG_INSTALL $DEPS

# Install td-agent
echo -e "${YELLOW}[4/8] Installing td-agent...${NC}"
curl -fsSL "https://toolbelt.treasuredata.com/sh/${INSTALL_SCRIPT}" | sh

# Create directories and set permissions
echo -e "${YELLOW}[5/8] Setting up directories and permissions...${NC}"
mkdir -p /var/log/td-agent/buffer /etc/td-agent/conf.d
chown -R td-agent:td-agent /var/log/td-agent
chmod 755 /var/log/td-agent
chmod 755 /var/log/td-agent/buffer
usermod -a -G adm td-agent

# Configure SELinux if present
if command -v getenforce >/dev/null 2>&1 && [ "$(getenforce)" != "Disabled" ]; then
    echo -e "${YELLOW}Configuring SELinux for td-agent...${NC}"
    setsebool -P httpd_can_network_connect 1 2>/dev/null || true
fi

# Create main configuration
echo -e "${YELLOW}[6/8] Creating td-agent configuration...${NC}"
cat > /etc/td-agent/td-agent.conf << 'EOF'
<system>
  log_level info
  suppress_repeated_stacktrace true
  emit_error_log_interval 30s
  suppress_config_dump
  without_source true
</system>

@include conf.d/*.conf

<source>
  @type tail
  @id syslog_input
  tag system.syslog
  path /var/log/syslog,/var/log/messages
  pos_file /var/log/td-agent/syslog.log.pos
  <parse>
    @type syslog
    message_format rfc3164
    with_priority true
  </parse>
</source>

<source>
  @type tail
  @id auth_input
  tag system.auth
  path /var/log/auth.log,/var/log/secure
  pos_file /var/log/td-agent/auth.log.pos
  <parse>
    @type syslog
    message_format rfc3164
    with_priority true
  </parse>
</source>

<filter **>
  @type record_transformer
  <record>
    hostname ${hostname}
    environment ENVIRONMENT_PLACEHOLDER
    log_source fluentd
  </record>
</filter>

<match system.**>
  @type file
  path /var/log/td-agent/system
  append true
  <format>
    @type json
  </format>
  <buffer time>
    timekey 1h
    timekey_use_utc true
    path /var/log/td-agent/buffer/system
  </buffer>
</match>

<match **>
  @type forward
  @id main_forward
  <server>
    name log-server
    host LOG_SERVER_PLACEHOLDER
    port 24224
    weight 60
  </server>
  <buffer>
    @type file
    path /var/log/td-agent/buffer/forward
    flush_mode interval
    flush_interval 5s
    flush_thread_count 2
    retry_type exponential_backoff
    retry_wait 1s
    retry_max_interval 60s
    retry_timeout 60m
    overflow_action block
  </buffer>
</match>
EOF

# Replace placeholders
sed -i "s/LOG_SERVER_PLACEHOLDER/$LOG_SERVER_HOST/g" /etc/td-agent/td-agent.conf
sed -i "s/ENVIRONMENT_PLACEHOLDER/$ENVIRONMENT/g" /etc/td-agent/td-agent.conf

# Create custom parser configuration
cat > /etc/td-agent/conf.d/custom-parsers.conf << 'EOF'
<source>
  @type tail
  @id app_json_input
  tag app.json
  path /var/log/app/*.json
  pos_file /var/log/td-agent/app.json.pos
  <parse>
    @type json
    time_key timestamp
    time_format %Y-%m-%dT%H:%M:%S.%L%z
  </parse>
</source>

<source>
  @type tail
  @id nginx_access_input
  tag web.nginx.access
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx.access.pos
  <parse>
    @type nginx
    format combined
  </parse>
</source>
EOF

# Set proper ownership and permissions
chown -R td-agent:td-agent /etc/td-agent
chmod 644 /etc/td-agent/td-agent.conf
chmod 644 /etc/td-agent/conf.d/*.conf
chmod 755 /etc/td-agent/conf.d

# Enable and start td-agent service
echo -e "${YELLOW}[7/8] Starting td-agent service...${NC}"
systemctl enable td-agent
systemctl start td-agent

# Wait for service to stabilize
sleep 5

# Verification checks
echo -e "${YELLOW}[8/8] Verifying installation...${NC}"

# Check service status
if systemctl is-active --quiet td-agent; then
    echo -e "${GREEN}✓ td-agent service is running${NC}"
else
    echo -e "${RED}✗ td-agent service is not running${NC}"
    exit 1
fi

# Check log files
if [ -f /var/log/td-agent/td-agent.log ]; then
    echo -e "${GREEN}✓ td-agent log file exists${NC}"
else
    echo -e "${RED}✗ td-agent log file not found${NC}"
    exit 1
fi

# Check configuration syntax
if td-agent --dry-run -c /etc/td-agent/td-agent.conf >/dev/null 2>&1; then
    echo -e "${GREEN}✓ Configuration syntax is valid${NC}"
else
    echo -e "${RED}✗ Configuration syntax error${NC}"
    exit 1
fi

# Check buffer directory permissions
if [ -w /var/log/td-agent/buffer ]; then
    echo -e "${GREEN}✓ Buffer directory is writable${NC}"
else
    echo -e "${RED}✗ Buffer directory is not writable${NC}"
    exit 1
fi

echo -e "${GREEN}✓ Fluentd (td-agent) installation completed successfully!${NC}"
echo -e "${YELLOW}Configuration details:${NC}"
echo "- Config file: /etc/td-agent/td-agent.conf"
echo "- Additional configs: /etc/td-agent/conf.d/"
echo "- Log files: /var/log/td-agent/"
echo "- Buffer directory: /var/log/td-agent/buffer/"
echo "- Log server: $LOG_SERVER_HOST:24224"
echo "- Environment: $ENVIRONMENT"
echo ""
echo -e "${YELLOW}Useful commands:${NC}"
echo "- Check status: systemctl status td-agent"
echo "- View logs: tail -f /var/log/td-agent/td-agent.log"
echo "- Restart service: systemctl restart td-agent"
echo "- Test config: td-agent --dry-run -c /etc/td-agent/td-agent.conf"

Review the script before running. Execute with: bash install.sh

#fluentd #log-collection #centralized-logging #td-agent #log-parsing

Install and configure Fluentd for centralized log collection with multi-format parsing

Prerequisites

What this solves

Step-by-step installation

Update system packages

Install required dependencies

Add Treasure Data repository

Create td-agent user and directories

Configure main td-agent configuration

Include additional configurations

Input: System logs

Input: Authentication logs

Input: Application logs with JSON format

Input: NGINX access logs

Filter: Add hostname and environment tags

Output: File for debugging

Output: Forward to centralized log server

Create custom parser for application logs

Multi-line parser for Java stack traces

CSV log parser

Configure multiple output destinations

Output critical errors to email alert

Output application logs to S3 for long-term storage

Configure SSL and security settings

Secure forward output with authentication

Create SSL certificates directory

Configure performance optimization

Buffer optimization for high throughput

Set up log rotation and maintenance

Enable and start td-agent service

Configure firewall rules

Verify your setup

Verify configuration syntax

Check td-agent version and plugins

Test log processing

Monitor real-time logs

Check buffer status

Test network connectivity to outputs

Performance tuning and monitoring

Install monitoring plugins

Configure monitoring endpoints

Debug agent for configuration inspection

Prometheus metrics endpoint

Internal metrics collection

Common issues

Next steps

Related tutorials

Configure Linux performance monitoring with collectd and InfluxDB 1.8 for real-time metrics collection

Set up Prometheus and Grafana monitoring stack with Docker compose

Install and configure Jaeger for distributed tracing with Elasticsearch backend

Don't want to manage this yourself?