Update system packages

Start by updating your package manager to ensure you get the latest versions.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install required packages

Install curl and unzip for downloading and extracting Caddy and Consul binaries.

sudo apt install -y curl unzip wget gpg

sudo dnf install -y curl unzip wget gpg

Install Consul

Download and install the latest Consul binary from HashiCorp's official releases. Consul will handle service discovery and health checking.

cd /tmp
wget https://releases.hashicorp.com/consul/1.17.0/consul_1.17.0_linux_amd64.zip
unzip consul_1.17.0_linux_amd64.zip
sudo mv consul /usr/local/bin/
sudo chmod +x /usr/local/bin/consul

Create Consul user and directories

Create a dedicated system user for Consul and set up necessary directories with proper permissions.

sudo useradd --system --home /etc/consul.d --shell /bin/false consul
sudo mkdir -p /etc/consul.d /opt/consul /var/lib/consul
sudo chown consul:consul /etc/consul.d /opt/consul /var/lib/consul
sudo chmod 755 /etc/consul.d /opt/consul /var/lib/consul

Configure Consul server

Create the main Consul configuration file with server settings, data directory, and client access configuration.

datacenter = "dc1"
data_dir = "/var/lib/consul"
log_level = "INFO"
server = true
bootstrap_expect = 1
bind_addr = "0.0.0.0"
client_addr = "0.0.0.0"
retry_join = ["127.0.0.1"]
ui_config {
  enabled = true
}
connect {
  enabled = true
}
ports {
  grpc = 8502
}
acl = {
  enabled = false
  default_policy = "allow"
}

Create Consul systemd service

Set up a systemd service file to manage Consul as a system service with automatic startup and proper security settings.

[Unit]
Description=Consul
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start and enable Consul

Start the Consul service and enable it to start automatically on boot.

sudo chown consul:consul /etc/consul.d/consul.hcl
sudo systemctl daemon-reload
sudo systemctl enable consul
sudo systemctl start consul
sudo systemctl status consul

Install Caddy

Install Caddy web server using the official installation script which adds the repository and GPG key.

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install -y caddy

dnf install -y 'dnf-command(copr)'
dnf copr enable -y @caddy/caddy
dnf install -y caddy

Configure Caddy with Consul service discovery

Create a Caddyfile that uses Consul for automatic service discovery and health checks. This configuration sets up dynamic upstream discovery for web services.

{
  admin localhost:2019
  order consul first
}

example.com {
  reverse_proxy {
    dynamic consul {
      service "web-service"
      refresh_interval 5s
    }
    health_uri /health
    health_interval 10s
    health_timeout 5s
  }
  
  log {
    output file /var/log/caddy/access.log
    format json
  }
}

api.example.com {
  reverse_proxy {
    dynamic consul {
      service "api-service"
      refresh_interval 5s
    }
    health_uri /api/health
    health_interval 10s
    health_timeout 5s
  }
}

Create log directory for Caddy

Create a log directory with proper permissions for Caddy to write access logs.

sudo mkdir -p /var/log/caddy
sudo chown caddy:caddy /var/log/caddy
sudo chmod 755 /var/log/caddy

Install Caddy Consul plugin

Download and install the Caddy binary with the Consul plugin for service discovery functionality.

cd /tmp
wget https://github.com/pteich/caddy-tlsconsul/releases/download/v0.2.0/caddy-consul-linux-amd64.tar.gz
tar -xzf caddy-consul-linux-amd64.tar.gz
sudo systemctl stop caddy
sudo cp caddy /usr/bin/caddy
sudo chmod 755 /usr/bin/caddy
sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/caddy

Configure firewall rules

Open necessary ports for Caddy (80, 443) and Consul (8500, 8600) communication.

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 8500/tcp
sudo ufw allow 8600/tcp
sudo ufw allow 8600/udp

sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
sudo firewall-cmd --permanent --add-port=8500/tcp
sudo firewall-cmd --permanent --add-port=8600/tcp
sudo firewall-cmd --permanent --add-port=8600/udp
sudo firewall-cmd --reload

Start Caddy service

Start Caddy and enable it to start automatically on boot. Verify the configuration is valid before starting.

sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl enable caddy
sudo systemctl start caddy
sudo systemctl status caddy

Register sample services with Consul

Register test services in Consul to demonstrate service discovery functionality. This creates web services that Caddy can automatically discover and route to.

{
  "ID": "web-service-1",
  "Name": "web-service",
  "Tags": ["web", "primary"],
  "Address": "203.0.113.10",
  "Port": 8080,
  "Check": {
    "HTTP": "http://203.0.113.10:8080/health",
    "Interval": "10s",
    "Timeout": "3s"
  }
}

{
  "ID": "web-service-2",
  "Name": "web-service",
  "Tags": ["web", "secondary"],
  "Address": "203.0.113.11",
  "Port": 8080,
  "Check": {
    "HTTP": "http://203.0.113.11:8080/health",
    "Interval": "10s",
    "Timeout": "3s"
  }
}

curl -X PUT --data @/tmp/web-service1.json http://127.0.0.1:8500/v1/agent/service/register
curl -X PUT --data @/tmp/web-service2.json http://127.0.0.1:8500/v1/agent/service/register

Configure health check endpoints

Add health check configuration to Caddy for monitoring backend service health and automatic failover.

{
  admin localhost:2019
  order consul first
  servers {
    metrics
  }
}

Main application load balancing
example.com {
  reverse_proxy {
    dynamic consul localhost:8500 {
      service web-service
      refresh_interval 30s
    }
    health_path /health
    health_interval 30s
    health_timeout 10s
    health_status 2xx
    
    # Load balancing policy
    lb_policy least_conn
    
    # Fail timeout
    fail_timeout 30s
    max_fails 3
  }
  
  # Enable access logging
  log {
    output file /var/log/caddy/example.com.log {
      roll_size 100MiB
      roll_keep 5
    }
    format json
  }
  
  # Enable compression
  encode gzip
}

API service load balancing
api.example.com {
  reverse_proxy {
    dynamic consul localhost:8500 {
      service api-service
      refresh_interval 30s
    }
    health_path /api/v1/health
    health_interval 30s
    health_timeout 10s
    
    lb_policy round_robin
    fail_timeout 30s
    max_fails 2
  }
}

Reload Caddy configuration

Apply the updated configuration and verify Caddy is running with the new settings.

sudo caddy reload --config /etc/caddy/Caddyfile
sudo systemctl status caddy

Add custom health check scripts

Create advanced health check scripts for more sophisticated service monitoring beyond simple HTTP checks.

#!/bin/bash
Advanced health check for web service
set -e

SERVICE_URL="http://localhost:8080"
HEALTH_ENDPOINT="${SERVICE_URL}/health"
METRICS_ENDPOINT="${SERVICE_URL}/metrics"

Check HTTP response
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$HEALTH_ENDPOINT" || echo "000")

if [ "$HTTP_CODE" -ne 200 ]; then
    echo "Health check failed: HTTP $HTTP_CODE"
    exit 1
fi

Check if service is responsive within 2 seconds
RESPONSE_TIME=$(curl -s -w "%{time_total}" -o /dev/null "$HEALTH_ENDPOINT")
if (( $(echo "$RESPONSE_TIME > 2.0" | bc -l) )); then
    echo "Service too slow: ${RESPONSE_TIME}s"
    exit 1
fi

Check metrics endpoint
METRICS_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$METRICS_ENDPOINT" || echo "000")
if [ "$METRICS_CODE" -ne 200 ]; then
    echo "Metrics endpoint failed: HTTP $METRICS_CODE"
    exit 1
fi

echo "Service healthy"
exit 0

sudo mkdir -p /opt/consul/health-checks
sudo chmod +x /opt/consul/health-checks/web-service-check.sh
sudo chown consul:consul /opt/consul/health-checks/web-service-check.sh

Register services with script-based checks

Update service definitions to use custom health check scripts for more comprehensive monitoring.

{
  "services": [
    {
      "id": "web-service-1",
      "name": "web-service",
      "tags": ["web", "primary", "v1.2.0"],
      "address": "203.0.113.10",
      "port": 8080,
      "meta": {
        "version": "1.2.0",
        "environment": "production"
      },
      "checks": [
        {
          "id": "web-http-check",
          "name": "HTTP API Check",
          "http": "http://203.0.113.10:8080/health",
          "method": "GET",
          "interval": "10s",
          "timeout": "3s",
          "deregister_critical_service_after": "30s"
        },
        {
          "id": "web-script-check",
          "name": "Advanced Health Check",
          "script": "/opt/consul/health-checks/web-service-check.sh",
          "interval": "30s",
          "timeout": "10s"
        }
      ]
    }
  ]
}

sudo chown consul:consul /etc/consul.d/web-services.json
sudo systemctl reload consul

Configure Caddy upstream selection

Set up advanced upstream selection policies and failover behavior in Caddy for high availability.

{
  admin localhost:2019
  order consul first
  servers {
    metrics
  }
}

Production traffic with intelligent failover
example.com {
  reverse_proxy {
    dynamic consul localhost:8500 {
      service web-service
      refresh_interval 10s
      
      # Only use healthy services
      health_check {
        uri /health
        interval 15s
        timeout 5s
        status 2xx
        body contains "ok"
      }
    }
    
    # Load balancing configuration
    lb_policy ip_hash  # Sticky sessions
    lb_try_duration 30s
    lb_try_interval 250ms
    
    # Health and failover
    health_uri /health
    health_interval 15s
    health_timeout 5s
    health_status 200
    
    # Failure handling
    fail_timeout 30s
    max_fails 3
    unhealthy_request_count 5
    
    # Passive health checking
    passive_health {
      unhealthy_status 5xx
      unhealthy_count 3
      unhealthy_duration 30s
      healthy_count 2
      healthy_duration 10s
    }
    
    # Circuit breaker pattern
    transport http {
      dial_timeout 5s
      response_header_timeout 10s
      expect_continue_timeout 1s
      keepalive_idle_conns 100
      keepalive_idle_conns_per_host 10
    }
  }
  
  # Error pages for upstream failures  
  handle_errors {
    @5xx expression {http.error.status_code} >= 500
    rewrite @5xx /errors/5xx.html
    file_server {
      root /var/www/errors
    }
  }
  
  log {
    output file /var/log/caddy/example.com.log {
      roll_size 100MiB
      roll_keep 10
    }
    format json {
      time_format "iso8601"
      message_key "msg"
    }
    level INFO
  }
  
  encode gzip zstd
}

Health check endpoint for load balancer
example.com/lb-health {
  respond "OK" 200 {
    headers {
      Content-Type "text/plain"
      X-Health-Check "caddy-consul"
    }
  }
}

Create error pages

Set up custom error pages for upstream failures to provide better user experience during outages.

sudo mkdir -p /var/www/errors
sudo chown caddy:caddy /var/www/errors

    
    
    


    
Service Temporarily Unavailable
    
We're experiencing technical difficulties. Please try again in a few minutes.
    
If the problem persists, please contact support.

Apply configuration and test failover

Reload Caddy with the new configuration and test the failover mechanism.

sudo caddy reload --config /etc/caddy/Caddyfile

Test failover by stopping one service
consul services deregister -id="web-service-1"

Check remaining services
consul health service web-service

Re-register the service
curl -X PUT --data @/tmp/web-service1.json http://127.0.0.1:8500/v1/agent/service/register