Configure Elasticsearch ILM for Automated Data Retention

Set up Elasticsearch ILM policies to automatically manage index lifecycles through hot, warm, cold, and delete phases. Reduce storage costs and optimize performance by automating data retention and storage tiering.

Prerequisites

Running Elasticsearch 7.0+ cluster
Cluster admin permissions
Basic understanding of Elasticsearch indices

What this solves

Elasticsearch Index Lifecycle Management (ILM) automatically manages your indices through different phases based on age, size, or document count. Without ILM, your indices grow indefinitely, consuming disk space and degrading search performance. ILM policies move indices through hot (active writes), warm (read-only), cold (infrequent access), and delete phases, optimizing storage costs and cluster performance.

Prerequisites and planning

Before configuring ILM policies, you need a running Elasticsearch cluster with proper node roles. Understanding your data patterns helps design effective policies.

Verify cluster health

Check your Elasticsearch cluster status and node configuration.

curl -X GET "localhost:9200/_cluster/health?pretty"
curl -X GET "localhost:9200/_cat/nodes?v&h=node.role,name"

Review existing indices

Examine current indices to understand size patterns and identify candidates for ILM management.

curl -X GET "localhost:9200/_cat/indices?v&s=store.size:desc"
curl -X GET "localhost:9200/_cat/indices?v&h=index,docs.count,store.size,creation.date.string"

Understanding ILM phases and policies

ILM manages indices through four distinct phases, each with specific actions and triggers.

Phase	Purpose	Common Actions	Typical Duration
Hot	Active indexing and frequent queries	Rollover, set priority	1-7 days
Warm	Read-only, less frequent queries	Allocate to warm nodes, force merge	7-30 days
Cold	Infrequent access, archived data	Allocate to cold nodes, freeze	30-90 days
Delete	Remove old data	Delete index	After retention period

Step-by-step ILM configuration

Create a basic ILM policy

Define a policy that manages indices through hot, warm, and delete phases based on age and size.

curl -X PUT "localhost:9200/_ilm/policy/logs-policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "5GB",
            "max_age": "1d",
            "max_docs": 10000000
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "number_of_replicas": 1,
            "include": {},
            "exclude": {},
            "require": {
              "data_tier": "warm"
            }
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

Create an advanced ILM policy with cold phase

Configure a comprehensive policy including cold phase for long-term archival before deletion.

curl -X PUT "localhost:9200/_ilm/policy/application-logs-policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "10GB",
            "max_age": "7d",
            "max_docs": 50000000
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "number_of_replicas": 1,
            "require": {
              "data_tier": "warm"
            }
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "set_priority": {
            "priority": 0
          },
          "allocate": {
            "number_of_replicas": 0,
            "require": {
              "data_tier": "cold"
            }
          },
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

Verify policy creation

List all ILM policies and examine the policy details to confirm proper configuration.

curl -X GET "localhost:9200/_ilm/policy?pretty"
curl -X GET "localhost:9200/_ilm/policy/logs-policy?pretty"

Applying ILM policies to indices and templates

Create an index template with ILM

Configure an index template that automatically applies ILM policy to new indices matching the pattern.

curl -X PUT "localhost:9200/_index_template/logs-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy",
      "index.lifecycle.rollover_alias": "logs"
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "text"
        },
        "level": {
          "type": "keyword"
        }
      }
    }
  },
  "priority": 200,
  "composed_of": [],
  "version": 1
}'

Create the initial index with alias

Set up the bootstrap index and alias for rollover functionality.

curl -X PUT "localhost:9200/logs-000001" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "logs": {
      "is_write_index": true
    }
  }
}'

Apply ILM policy to existing indices

Attach ILM policies to existing indices that weren't created with templates.

curl -X PUT "localhost:9200/existing-logs-*/_settings" -H 'Content-Type: application/json' -d'
{
  "index.lifecycle.name": "logs-policy"
}'

Monitoring and managing ILM policy execution

Monitor ILM execution status

Check the lifecycle status and phase progression of managed indices.

curl -X GET "localhost:9200/_ilm/status?pretty"
curl -X GET "localhost:9200/logs-*/_ilm/explain?pretty"

Force policy execution for testing

Manually trigger ILM actions to test policy behavior without waiting for timers.

curl -X POST "localhost:9200/_ilm/move/logs-000001" -H 'Content-Type: application/json' -d'
{
  "current_step": {
    "phase": "hot",
    "action": "complete",
    "name": "complete"
  },
  "next_step": {
    "phase": "warm",
    "action": "complete",
    "name": "complete"
  }
}'

Create monitoring queries

Set up queries to track ILM policy effectiveness and identify issues.

curl -X GET "localhost:9200/_cat/indices?v&h=index,health,status,pri,rep,docs.count,store.size,creation.date.string" | grep logs
curl -X GET "localhost:9200/_ilm/explain/logs-*?only_errors=true&pretty"

Advanced ILM configurations

Configure size-based rollover

Create a policy focused on index size rather than age for high-volume applications.

curl -X PUT "localhost:9200/_ilm/policy/metrics-policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "2GB",
            "max_docs": 25000000
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "readonly": {},
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "delete": {
        "min_age": "14d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

Set up conditional phase transitions

Configure policies that use multiple conditions for phase transitions.

curl -X PUT "localhost:9200/_ilm/policy/conditional-policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "5GB",
            "max_age": "2d",
            "max_docs": 10000000
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "allocate": {
            "number_of_replicas": 1,
            "include": {},
            "exclude": {
              "_tier_preference": "data_hot"
            }
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "wait_for_snapshot": {
            "policy": "nightly-snapshots"
          },
          "delete": {}
        }
      }
    }
  }
}'

Verify your setup

curl -X GET "localhost:9200/_ilm/policy?pretty" | jq '.'
curl -X GET "localhost:9200/_cat/indices?v&s=creation.date" | head -10
curl -X GET "localhost:9200/logs-*/_ilm/explain?pretty" | jq '.indices[] | {index: .index, phase: .phase, action: .action}'
curl -X GET "localhost:9200/_cluster/health?pretty"

Note: If you're setting up monitoring for your ILM policies, consider integrating with Prometheus and Grafana to track index lifecycle metrics and storage utilization.

Common issues

Symptom	Cause	Fix
Policy not executing	ILM disabled or indices missing policy	Check `/_ilm/status` and verify policy assignment
Rollover not triggering	Write index alias not set	Ensure alias has `is_write_index: true`
Phase transitions failing	Node allocation requirements not met	Verify data tier nodes exist with proper attributes
Indices stuck in error state	Conflicting shard allocation rules	Check `/_ilm/explain?only_errors=true` for details
Unexpected storage growth	Delete phase not configured or delayed	Review policy timing and force execution for testing
Performance degradation	Too many small indices or segments	Adjust rollover thresholds and force merge settings

Next steps

Set up Elasticsearch cross-cluster replication for disaster recovery of managed indices
Monitor your Elasticsearch cluster to track ILM policy effectiveness
Configure snapshot lifecycle management to backup indices before deletion
Implement hot-warm-cold architecture for advanced data tiering strategies

Running this in production?

Want this handled for you? Setting up ILM policies once is straightforward. Keeping them tuned, monitored, and optimized for changing data patterns across environments is the harder part. See how we run infrastructure like this for European SaaS and e-commerce teams.

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash

set -euo pipefail

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

# Default values
ELASTICSEARCH_HOST="${1:-localhost}"
ELASTICSEARCH_PORT="${2:-9200}"
RETENTION_DAYS="${3:-30}"
HOT_SIZE="${4:-5GB}"
HOT_AGE="${5:-1d}"

usage() {
    echo "Usage: $0 [elasticsearch_host] [elasticsearch_port] [retention_days] [hot_size] [hot_age]"
    echo "Example: $0 localhost 9200 30 5GB 1d"
    echo "Defaults: localhost 9200 30 5GB 1d"
    exit 1
}

cleanup() {
    echo -e "${RED}[ERROR] Script failed. Check Elasticsearch logs for details.${NC}"
}

trap cleanup ERR

print_step() {
    echo -e "${BLUE}[$1] $2${NC}"
}

print_success() {
    echo -e "${GREEN}[SUCCESS] $1${NC}"
}

print_warning() {
    echo -e "${YELLOW}[WARNING] $1${NC}"
}

print_error() {
    echo -e "${RED}[ERROR] $1${NC}"
}

# Detect distribution
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian) 
            PKG_MGR="apt"
            PKG_INSTALL="apt install -y"
            PKG_UPDATE="apt update"
            ES_CONFIG_DIR="/etc/elasticsearch"
            ES_SERVICE="elasticsearch"
            ;;
        almalinux|rocky|centos|rhel|ol)
            PKG_MGR="dnf"
            PKG_INSTALL="dnf install -y"
            PKG_UPDATE="dnf update -y"
            ES_CONFIG_DIR="/etc/elasticsearch"
            ES_SERVICE="elasticsearch"
            ;;
        fedora)
            PKG_MGR="dnf"
            PKG_INSTALL="dnf install -y"
            PKG_UPDATE="dnf update -y"
            ES_CONFIG_DIR="/etc/elasticsearch"
            ES_SERVICE="elasticsearch"
            ;;
        amzn)
            PKG_MGR="yum"
            PKG_INSTALL="yum install -y"
            PKG_UPDATE="yum update -y"
            ES_CONFIG_DIR="/etc/elasticsearch"
            ES_SERVICE="elasticsearch"
            ;;
        *)
            print_error "Unsupported distribution: $ID"
            exit 1
            ;;
    esac
else
    print_error "Cannot detect distribution. /etc/os-release not found."
    exit 1
fi

# Check if running as root or with sudo
if [[ $EUID -ne 0 ]]; then
    print_error "This script must be run as root or with sudo"
    exit 1
fi

print_step "1/10" "Installing prerequisites..."
$PKG_UPDATE
$PKG_INSTALL curl jq

print_step "2/10" "Checking Elasticsearch connectivity..."
if ! curl -s "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_cluster/health" > /dev/null; then
    print_error "Cannot connect to Elasticsearch at ${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"
    print_warning "Please ensure Elasticsearch is running and accessible"
    exit 1
fi

print_success "Connected to Elasticsearch"

print_step "3/10" "Checking cluster health..."
CLUSTER_STATUS=$(curl -s "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_cluster/health" | jq -r '.status')
echo "Cluster status: $CLUSTER_STATUS"

if [ "$CLUSTER_STATUS" = "red" ]; then
    print_warning "Cluster status is RED. Consider fixing cluster issues before proceeding."
fi

print_step "4/10" "Reviewing existing indices..."
curl -s "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_cat/indices?v&s=store.size:desc" | head -10

print_step "5/10" "Creating basic logs ILM policy..."
curl -X PUT "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_ilm/policy/logs-policy" \
  -H 'Content-Type: application/json' -d"{
  \"policy\": {
    \"phases\": {
      \"hot\": {
        \"min_age\": \"0ms\",
        \"actions\": {
          \"rollover\": {
            \"max_size\": \"${HOT_SIZE}\",
            \"max_age\": \"${HOT_AGE}\",
            \"max_docs\": 10000000
          },
          \"set_priority\": {
            \"priority\": 100
          }
        }
      },
      \"warm\": {
        \"min_age\": \"${HOT_AGE}\",
        \"actions\": {
          \"set_priority\": {
            \"priority\": 50
          },
          \"allocate\": {
            \"number_of_replicas\": 1
          },
          \"forcemerge\": {
            \"max_num_segments\": 1
          }
        }
      },
      \"delete\": {
        \"min_age\": \"${RETENTION_DAYS}d\",
        \"actions\": {
          \"delete\": {}
        }
      }
    }
  }
}"

print_success "Basic logs policy created"

print_step "6/10" "Creating advanced application logs ILM policy..."
curl -X PUT "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_ilm/policy/application-logs-policy" \
  -H 'Content-Type: application/json' -d"{
  \"policy\": {
    \"phases\": {
      \"hot\": {
        \"min_age\": \"0ms\",
        \"actions\": {
          \"rollover\": {
            \"max_size\": \"10GB\",
            \"max_age\": \"7d\",
            \"max_docs\": 50000000
          },
          \"set_priority\": {
            \"priority\": 100
          }
        }
      },
      \"warm\": {
        \"min_age\": \"7d\",
        \"actions\": {
          \"set_priority\": {
            \"priority\": 50
          },
          \"allocate\": {
            \"number_of_replicas\": 1
          },
          \"forcemerge\": {
            \"max_num_segments\": 1
          }
        }
      },
      \"cold\": {
        \"min_age\": \"30d\",
        \"actions\": {
          \"set_priority\": {
            \"priority\": 0
          },
          \"allocate\": {
            \"number_of_replicas\": 0
          }
        }
      },
      \"delete\": {
        \"min_age\": \"90d\",
        \"actions\": {
          \"delete\": {}
        }
      }
    }
  }
}"

print_success "Advanced application logs policy created"

print_step "7/10" "Creating metrics ILM policy..."
curl -X PUT "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_ilm/policy/metrics-policy" \
  -H 'Content-Type: application/json' -d'{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "2GB",
            "max_age": "1d",
            "max_docs": 5000000
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "set_priority": {
            "priority": 50
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "delete": {
        "min_age": "14d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

print_success "Metrics policy created"

print_step "8/10" "Creating index templates with ILM policies..."
curl -X PUT "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_index_template/logs-template" \
  -H 'Content-Type: application/json' -d'{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-policy",
      "index.lifecycle.rollover_alias": "logs",
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  }
}'

curl -X PUT "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_index_template/metrics-template" \
  -H 'Content-Type: application/json' -d'{
  "index_patterns": ["metrics-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "metrics-policy",
      "index.lifecycle.rollover_alias": "metrics",
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  }
}'

print_success "Index templates created"

print_step "9/10" "Verifying ILM policies..."
echo "Created ILM policies:"
curl -s "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_ilm/policy" | jq -r 'keys[]' | grep -E "(logs-policy|application-logs-policy|metrics-policy)"

print_step "10/10" "Final verification and status..."
echo "ILM Status:"
curl -s "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_ilm/status" | jq '.'

echo "Policy Details:"
curl -s "http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_ilm/policy/logs-policy" | jq '.logs-policy.policy.phases | keys'

print_success "Elasticsearch ILM configuration completed successfully!"
echo ""
echo "Summary of configured policies:"
echo "- logs-policy: ${HOT_SIZE} hot size, ${HOT_AGE} hot age, ${RETENTION_DAYS}d retention"
echo "- application-logs-policy: 10GB hot size, 7d hot age, 90d retention with cold phase"
echo "- metrics-policy: 2GB hot size, 1d hot age, 14d retention"
echo ""
echo "To apply policies to existing indices, use:"
echo "curl -X PUT \"http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/your-index/_settings\" -H 'Content-Type: application/json' -d '{\"index.lifecycle.name\": \"logs-policy\"}'"

Review the script before running. Execute with: bash install.sh

#elasticsearch #ilm #data-retention #index-management #storage-optimization

Configure Elasticsearch Index Lifecycle Management (ILM) for automated data retention