Implement ScyllaDB disaster recovery with cross-region replication

Infrastructure requirements

You'll need at least 6 servers across two regions (minimum 3 nodes per region) with sufficient network bandwidth between regions for replication traffic.

Network and firewall configuration

Configure firewall rules to allow ScyllaDB cluster communication and monitoring access across regions.

sudo ufw allow 7000/tcp  # Inter-node communication
sudo ufw allow 7001/tcp  # TLS inter-node communication
sudo ufw allow 9042/tcp  # CQL native transport
sudo ufw allow 9160/tcp  # Thrift RPC
sudo ufw allow 10000/tcp # REST API
sudo ufw allow 9180/tcp  # Prometheus metrics
sudo ufw allow 19042/tcp # CQL shard-aware port

Install ScyllaDB on all nodes

Install ScyllaDB packages and dependencies on every cluster node across both regions.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5e08fbd8b5d6ec9c
sudo curl -L --output /etc/apt/sources.list.d/scylla.list http://downloads.scylladb.com/deb/ubuntu/scylla-5.4.list
sudo apt update
sudo apt install -y scylla

sudo curl -L --output /etc/yum.repos.d/scylla.repo http://downloads.scylladb.com/rpm/centos/scylla-5.4.repo
sudo dnf install -y scylla
sudo dnf install -y python3-PyYAML

Configure system optimization

Run ScyllaDB's setup script to optimize system parameters for maximum performance and reliability.

sudo scylla_setup --no-raid-setup
sudo scylla_io_setup

Configure ScyllaDB for multi-region topology

Create the main configuration file with topology-aware settings and cross-region replication support.

# Cluster settings
cluster_name: 'production-cluster'
num_tokens: 256
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer

# Network settings
listen_address: 10.0.1.10  # Replace with actual node IP
broadcast_address: 10.0.1.10  # Replace with actual node IP
rpc_address: 0.0.0.0
broadcast_rpc_address: 10.0.1.10  # Replace with actual node IP

# Seeds configuration (include nodes from both regions)
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "10.0.1.10,10.0.1.11,10.0.1.12,10.0.2.10,10.0.2.11,10.0.2.12"

# Topology settings
endpoint_snitch: GossipingPropertyFileSnitch
data_file_directories:
    - /var/lib/scylla/data
commitlog_directory: /var/lib/scylla/commitlog
hints_directory: /var/lib/scylla/hints
view_hints_directory: /var/lib/scylla/view_hints

# Cross-region optimization
inter_dc_stream_throughput_outbound_megabits_per_sec: 1000
inter_dc_tcp_nodelay: true
read_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 10000
range_request_timeout_in_ms: 20000

# Enable experimental features for better replication
experimental_features:
    - udf
    - alternator-streams

# Backup and snapshot settings
snapshot_before_compaction: false
auto_snapshot: true
incremental_backups: true

Configure datacenter topology

Define datacenter and rack information for proper replica placement and network topology awareness.

# Region 1 nodes (adjust for each node)
dc=us-east-1
rack=rack1

# For region 2 nodes, use:
# dc=us-west-1
</code><h2><code>rack=rack1</code></h2></pre>

<div class="info"><strong>Note:</strong> Each node must have the correct datacenter and rack configuration before starting. Update these values on each node accordingly.</div>
</div>

<div class="step">
### Start ScyllaDB services
<p>Enable and start ScyllaDB on all nodes, beginning with seed nodes in the first region.</p>

<pre class="terminal"><code>sudo systemctl enable scylla-server
sudo systemctl start scylla-server
sudo systemctl status scylla-server

Verify cluster formation

Check that all nodes have joined the cluster and can communicate across regions.

nodetool status
nodetool ring
nodetool describecluster

Create keyspace with NetworkTopologyStrategy

Create keyspaces configured for multi-datacenter replication with appropriate consistency levels.

cqlsh -u cassandra -p cassandra

CREATE KEYSPACE production_data 
WITH replication = {
    'class': 'NetworkTopologyStrategy',
    'us-east-1': 3,
    'us-west-1': 3
};

CREATE KEYSPACE system_auth_backup 
WITH replication = {
    'class': 'NetworkTopologyStrategy',
    'us-east-1': 3,
    'us-west-1': 3
};

USE production_data;

CREATE TABLE user_sessions (
    session_id UUID PRIMARY KEY,
    user_id UUID,
    created_at TIMESTAMP,
    last_access TIMESTAMP,
    session_data TEXT
) WITH gc_grace_seconds = 86400;

Configure consistency levels

Set appropriate consistency levels for read and write operations to balance performance with data consistency.

-- For strong consistency across regions
CONSISTENCY EACH_QUORUM;

-- For local datacenter consistency (better performance)
CONSISTENCY LOCAL_QUORUM;

-- Test consistency settings
SELECT * FROM system.local;
SELECT * FROM system.peers;

Install backup dependencies

Install required tools for automated backup creation, compression, and remote storage.

sudo apt install -y awscli s3cmd pigz parallel
pip3 install --user scylla-manager-client

sudo dnf install -y awscli s3cmd pigz parallel
pip3 install --user scylla-manager-client

Create backup automation script

Develop a comprehensive backup script that handles snapshots, compression, and remote storage with error handling.

#!/bin/bash

# ScyllaDB Automated Backup Script
set -euo pipefail

# Configuration
BACKUP_DIR="/opt/scylla/backups"
S3_BUCKET="s3://scylla-backups-production"
RETENTION_DAYS=30
LOG_FILE="/var/log/scylla/backup.log"
DATE=$(date +%Y%m%d_%H%M%S)
HOSTNAME=$(hostname -f)

# Logging function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

log "Starting backup process on $HOSTNAME"

# Create backup directory
mkdir -p "$BACKUP_DIR/$DATE"

# Create snapshot
log "Creating snapshot"
nodetool snapshot -t "backup_$DATE"

# Find and compress snapshot files
log "Compressing snapshot data"
find /var/lib/scylla/data -name "backup_$DATE" -type d | while read -r snapshot_dir; do
    keyspace=$(echo "$snapshot_dir" | cut -d'/' -f6)
    table=$(echo "$snapshot_dir" | cut -d'/' -f7)
    
    # Create compressed archive
    tar -I pigz -cf "$BACKUP_DIR/$DATE/${keyspace}_${table}_$DATE.tar.gz" -C "$snapshot_dir" .
    
    log "Compressed $keyspace.$table"
done

# Upload to S3
log "Uploading to S3"
aws s3 sync "$BACKUP_DIR/$DATE" "$S3_BUCKET/$HOSTNAME/$DATE/" --storage-class STANDARD_IA

# Cleanup old local backups
log "Cleaning up old local backups"
find "$BACKUP_DIR" -type d -mtime +7 -exec rm -rf {} +

# Cleanup old snapshots
log "Cleaning up old snapshots"
nodetool clearsnapshot

# Cleanup old S3 backups
log "Cleaning up old S3 backups"
aws s3 ls "$S3_BUCKET/$HOSTNAME/" | while read -r line; do
    backup_date=$(echo "$line" | awk '{print $2}' | tr -d '/')
    if [[ $(date -d "$backup_date" +%s 2>/dev/null || echo 0) -lt $(date -d "$RETENTION_DAYS days ago" +%s) ]]; then
        aws s3 rm "$S3_BUCKET/$HOSTNAME/$backup_date/" --recursive
        log "Removed old backup: $backup_date"
    fi
done

log "Backup process completed successfully"

# Health check
nodetool status | grep -q "UN" && log "Cluster health: OK" || log "Cluster health: WARNING"

sudo chmod +x /opt/scylla/backup-automation.sh
sudo mkdir -p /var/log/scylla

Configure S3 credentials

Set up AWS credentials and S3 bucket configuration for secure backup storage.

sudo mkdir -p /root/.aws

[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
region = us-east-1

[default]
region = us-east-1
output = json

[profile backup]
region = us-east-1
s3 =
    max_concurrent_requests = 20
    max_bandwidth = 100MB/s

Schedule automated backups

Configure systemd timers for reliable backup scheduling with proper error handling and monitoring.

[Unit]
Description=ScyllaDB Automated Backup
After=scylla-server.service
Requires=scylla-server.service

[Service]
Type=oneshot
User=root
Group=root
ExecStart=/opt/scylla/backup-automation.sh
StandardOutput=journal
StandardError=journal

[Unit]
Description=Run ScyllaDB backup daily at 2 AM
Requires=scylla-backup.service

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=300
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable scylla-backup.timer
sudo systemctl start scylla-backup.timer
sudo systemctl status scylla-backup.timer

Install monitoring components

Install Prometheus and Grafana for comprehensive ScyllaDB cluster monitoring with custom dashboards.

sudo apt install -y prometheus prometheus-node-exporter grafana
wget https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-4.6.tar.gz
tar -xzf scylla-monitoring-4.6.tar.gz
sudo mv scylla-monitoring-4.6 /opt/scylla-monitoring

sudo dnf install -y prometheus prometheus-node-exporter grafana
wget https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-4.6.tar.gz
tar -xzf scylla-monitoring-4.6.tar.gz
sudo mv scylla-monitoring-4.6 /opt/scylla-monitoring

Configure Prometheus for ScyllaDB

Set up Prometheus configuration to scrape metrics from all ScyllaDB nodes across regions.

# ScyllaDB cluster nodes
- targets:
  - 10.0.1.10:9180
  - 10.0.1.11:9180
  - 10.0.1.12:9180
  labels:
    cluster: production-cluster
    dc: us-east-1
    
- targets:
  - 10.0.2.10:9180
  - 10.0.2.11:9180
  - 10.0.2.12:9180
  labels:
    cluster: production-cluster
    dc: us-west-1

# Node exporter targets
- targets:
  - 10.0.1.10:9100
  - 10.0.1.11:9100
  - 10.0.1.12:9100
  - 10.0.2.10:9100
  - 10.0.2.11:9100
  - 10.0.2.12:9100

Create failover automation script

Develop intelligent failover automation that detects datacenter failures and redirects application traffic.

#!/bin/bash

# ScyllaDB Failover Automation Script
set -euo pipefail

# Configuration
PRIMARY_DC="us-east-1"
SECONDARY_DC="us-west-1"
PRIMARY_NODES=("10.0.1.10" "10.0.1.11" "10.0.1.12")
SECONDARY_NODES=("10.0.2.10" "10.0.2.11" "10.0.2.12")
LOAD_BALANCER_CONFIG="/etc/haproxy/haproxy.cfg"
ALERT_EMAIL="ops@example.com"
LOG_FILE="/var/log/scylla/failover.log"

# Logging
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

# Health check function
check_datacenter_health() {
    local dc=$1
    local nodes=()
    local healthy_count=0
    
    if [[ "$dc" == "$PRIMARY_DC" ]]; then
        nodes=("${PRIMARY_NODES[@]}")
    else
        nodes=("${SECONDARY_NODES[@]}")
    fi
    
    for node in "${nodes[@]}"; do
        if timeout 5 cqlsh "$node" -e "SELECT now() FROM system.local" &>/dev/null; then
            ((healthy_count++))
        fi
    done
    
    # Require majority of nodes to be healthy
    if [[ $healthy_count -ge 2 ]]; then
        return 0  # Healthy
    else
        return 1  # Unhealthy
    fi
}

# Update load balancer configuration
update_load_balancer() {
    local active_dc=$1
    
    log "Updating load balancer to use $active_dc"
    
    # Generate new HAProxy config
    cat > "$LOAD_BALANCER_CONFIG" << EOF
global
    daemon
    log stdout local0
    
defaults
    mode tcp
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    
listen scylla-cluster
    bind *:9042
    balance roundrobin
    option tcp-check
    tcp-check send-binary 040000000a000400436040
    tcp-check expect binary 8400000a
EOF
    
    if [[ "$active_dc" == "$PRIMARY_DC" ]]; then
        for node in "${PRIMARY_NODES[@]}"; do
            echo "    server node-$node $node:9042 check" >> "$LOAD_BALANCER_CONFIG"
        done
    else
        for node in "${SECONDARY_NODES[@]}"; do
            echo "    server node-$node $node:9042 check" >> "$LOAD_BALANCER_CONFIG"
        done
    fi
    
    systemctl reload haproxy
}

# Send alert
send_alert() {
    local message=$1
    echo "$message" | mail -s "ScyllaDB Failover Alert" "$ALERT_EMAIL"
    log "Alert sent: $message"
}

# Main failover logic
log "Starting failover check"

if check_datacenter_health "$PRIMARY_DC"; then
    log "Primary datacenter ($PRIMARY_DC) is healthy"
    # Ensure we're using primary DC
    if ! grep -q "${PRIMARY_NODES[0]}" "$LOAD_BALANCER_CONFIG"; then
        log "Failing back to primary datacenter"
        update_load_balancer "$PRIMARY_DC"
        send_alert "ScyllaDB: Failed back to primary datacenter $PRIMARY_DC"
    fi
else
    log "Primary datacenter ($PRIMARY_DC) is unhealthy"
    
    if check_datacenter_health "$SECONDARY_DC"; then
        log "Secondary datacenter ($SECONDARY_DC) is healthy, initiating failover"
        update_load_balancer "$SECONDARY_DC"
        send_alert "ScyllaDB: Failed over to secondary datacenter $SECONDARY_DC"
    else
        log "Both datacenters are unhealthy - CRITICAL ALERT"
        send_alert "CRITICAL: ScyllaDB cluster completely unavailable - both datacenters down"
    fi
fi

log "Failover check completed"

sudo chmod +x /opt/scylla/failover-automation.sh

Configure automated failover monitoring

Set up continuous monitoring with automatic failover detection and execution.

[Unit]
Description=ScyllaDB Failover Monitor
After=network.target scylla-server.service

[Service]
Type=oneshot
User=root
Group=root
ExecStart=/opt/scylla/failover-automation.sh
StandardOutput=journal
StandardError=journal

[Unit]
Description=Run ScyllaDB failover check every 30 seconds
Requires=scylla-failover.service

[Timer]
OnCalendar=*:*:0/30
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable scylla-failover.timer
sudo systemctl start scylla-failover.timer

Set up Grafana dashboards

Configure comprehensive monitoring dashboards for cluster health, performance metrics, and disaster recovery status.

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

# Import ScyllaDB monitoring dashboards
cd /opt/scylla-monitoring
sudo ./start-all.sh -d /opt/scylla-monitoring/prometheus/scylla_servers.yml -s /opt/scylla-monitoring/prometheus/scylla_manager_servers.yml

Configure alerting rules

Set up Prometheus alerting rules for critical Sc