How to Configure Elasticsearch Index Lifecycle Management (ILM) with LogSentinelAI

LogSentinelAI automatically creates an ILM policy named logsentinelai-analysis-policy that rolls over indices at 10GB or 1 day and deletes them after 7 days, binding it to indices via the logsentinelai-analysis-template index template.

LogSentinelAI is an open-source log analysis tool that enriches security events and stores them in Elasticsearch. To manage storage costs and compliance requirements, you need to configure Elasticsearch Index Lifecycle Management (ILM) with LogSentinelAI to automate index rollovers and deletions. This guide walks through the exact implementation found in the call518/logsentinelai repository.

Understanding LogSentinelAI ILM Architecture

LogSentinelAI delegates all index lifecycle decisions to Elasticsearch through a dedicated policy and template. The application code never manages retention logic directly; instead, it writes to a rollover alias and lets Elasticsearch handle the rest.

Core Components and File Locations

• src/logsentinelai/core/elasticsearch.py – Establishes the connection using ELASTICSEARCH_HOST, ELASTICSEARCH_USER, and ELASTICSEARCH_PASSWORD, then writes documents to the index specified by ELASTICSEARCH_INDEX.

• src/logsentinelai/core/config.py – Defines default connection parameters including the host (http://localhost:9200) and the default index prefix logsentinelai-analysis.

• INSTALL-and-USAGE.md – Contains the exact curl commands required to create the ILM policy and index template.

• Wiki/Home.md – Documents the default retention phases (hot 7 days, warm 30 days, cold 90 days, delete 365 days) for long-term compliance strategies.

Creating the ILM Policy for LogSentinelAI

Before running LogSentinelAI, you must create the ILM policy that defines when indices roll over and when they are deleted. The default policy, logsentinelai-analysis-policy, is designed for high-volume security logs.

The policy defines two phases:

• Hot phase – Rolls over the index when it reaches 10GB or 1 day, whichever comes first.
• Delete phase – Permanently removes the index after 7 days from rollover.

Execute this command against your Elasticsearch cluster:

curl -X PUT "localhost:9200/_ilm/policy/logsentinelai-analysis-policy" \
     -H "Content-Type: application/json" \
     -u elastic:changeme \
     -d '{
       "policy": {
         "phases": {
           "hot": {
             "actions": {
               "rollover": {
                 "max_size": "10gb",
                 "max_age": "1d"
               }
             }
           },
           "delete": {
             "min_age": "7d",
             "actions": { "delete": {} }
           }
         }
       }
     }'

Configuring the Index Template

After creating the policy, you must bind it to the actual indices using an index template. LogSentinelAI uses the template logsentinelai-analysis-template to ensure every new index automatically inherits the ILM settings.

The template performs three critical functions:

1. Pattern matching – Applies to all indices named logsentinelai-analysis-*.
2. Lifecycle binding – Sets index.lifecycle.name to logsentinelai-analysis-policy and defines the rollover alias as logsentinelai-analysis.
3. Mapping configuration – Defines the schema for enriched log fields, including geo_point mappings for GeoIP data.

Run this command to create the template:

curl -X PUT "localhost:9200/_index_template/logsentinelai-analysis-template" \
     -H "Content-Type: application/json" \
     -u elastic:changeme \
     -d '{
       "index_patterns": ["logsentinelai-analysis-*"],
       "template": {
         "settings": {
           "number_of_shards": 1,
           "number_of_replicas": 1,
           "index.lifecycle.name": "logsentinelai-analysis-policy",
           "index.lifecycle.rollover_alias": "logsentinelai-analysis",
           "index.mapping.total_fields.limit": "10000"
         },
         "mappings": {
           "properties": {
             "timestamp": { "type": "date" },
             "source_ip": { "type": "ip" },
             "geo_location": { "type": "geo_point" }
           }
         }
       }
     }'

Integrating ILM with LogSentinelAI Runtime

Once the policy and template exist, LogSentinelAI automatically handles the rest. The application writes to the rollover alias logsentinelai-analysis, and Elasticsearch manages the underlying indices according to your ILM rules.

Environment Configuration

Override defaults using environment variables defined in src/logsentinelai/core/config.py:

export ELASTICSEARCH_HOST="https://es-cluster.example.com:9200"
export ELASTICSEARCH_USER="logsentinel"
export ELASTICSEARCH_PASSWORD="secure-password"
export ELASTICSEARCH_INDEX="logsentinelai-analysis"

Running with Elasticsearch Output

Execute the CLI with --output elasticsearch to stream enriched logs directly to the managed indices:

logsentinelai-httpd-access /var/log/apache2/access.log \
    --output elasticsearch \
    --mode realtime

Each batch of analyzed logs is indexed into the current write index (e.g., logsentinelai-analysis-000001). When the index reaches 10GB or 1 day of age, Elasticsearch automatically rolls over to logsentinelai-analysis-000002 and begins the lifecycle countdown for the previous index.

Customizing ILM Retention Policies

The default 7-day retention suits development environments, but production security operations often require longer retention for compliance. You can modify the ILM policy without touching the LogSentinelAI codebase.

To extend retention to 30 days, update the min_age in the delete phase:

curl -X PUT "localhost:9200/_ilm/policy/logsentinelai-analysis-policy" \
     -H "Content-Type: application/json" \
     -u elastic:changeme \
     -d '{
       "policy": {
         "phases": {
           "hot": {
             "actions": {
               "rollover": {
                 "max_size": "10gb",
                 "max_age": "1d"
               }
             }
           },
           "delete": {
             "min_age": "30d",
             "actions": { "delete": {} }
           }
         }
       }
     }'

Changes take effect immediately for new rollovers. Existing indices continue with the policy version they were created under unless you manually trigger a retry via the Elasticsearch ILM API.

Summary

• LogSentinelAI delegates index lifecycle management entirely to Elasticsearch ILM, writing to the rollover alias logsentinelai-analysis defined in src/logsentinelai/core/config.py.
• You must create the ILM policy logsentinelai-analysis-policy and index template logsentinelai-analysis-template via the curl commands documented in INSTALL-and-USAGE.md before running the application.
• The default configuration rolls indices at 10GB or 1 day and deletes them after 7 days, but you can customize retention by updating the policy JSON without restarting LogSentinelAI.
• Environment variables (ELASTICSEARCH_HOST, ELASTICSEARCH_USER, ELASTICSEARCH_PASSWORD, ELASTICSEARCH_INDEX) allow you to point the CLI at any Elasticsearch cluster without code changes.

Frequently Asked Questions

What is the default ILM retention period in LogSentinelAI?

The default ILM policy created for LogSentinelAI retains indices for 7 days after rollover. According to the Wiki documentation in Wiki/Home.md, broader organizational strategies may extend this to hot (7 days), warm (30 days), cold (90 days), and delete (365 days) phases, but the initial setup script configures a simple hot-to-delete transition at 7 days.

How do I change the rollover threshold from 10GB to a different size?

Modify the max_size parameter in the hot phase actions of the ILM policy. Send a PUT request to /_ilm/policy/logsentinelai-analysis-policy with an updated JSON payload containing "max_size": "50gb" (or your desired threshold). The change applies to the next rollover; current write indices continue with their existing thresholds.

Can I use a custom index name instead of logsentinelai-analysis?

Yes. Set the ELASTICSEARCH_INDEX environment variable to your preferred alias name before starting LogSentinelAI. You must also update the index_patterns in your index template and the rollover_alias in both the template and ILM policy to match your custom name. The configuration loader in src/logsentinelai/core/config.py reads these variables at startup.

Where does LogSentinelAI store the ILM policy configuration?

LogSentinelAI does not store the ILM policy internally; it resides entirely within your Elasticsearch cluster under the policy name logsentinelai-analysis-policy. The application references this policy through the index template setting index.lifecycle.name defined in INSTALL-and-USAGE.md. To inspect or modify the policy, use the Elasticsearch REST API at /_ilm/policy/logsentinelai-analysis-policy.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}