# How to Configure Elasticsearch Index Lifecycle Management (ILM) with LogSentinelAI

> Learn how to configure Elasticsearch Index Lifecycle Management ILM with LogSentinelAI. Discover automatic policy creation, index rollover, and data deletion for efficient log management.

- Repository: [JungJungIn/logsentinelai](https://github.com/call518/logsentinelai)
- Tags: how-to-guide
- Published: 2026-02-26

---

**LogSentinelAI automatically creates an ILM policy named `logsentinelai-analysis-policy` that rolls over indices at 10GB or 1 day and deletes them after 7 days, binding it to indices via the `logsentinelai-analysis-template` index template.**

LogSentinelAI is an open-source log analysis tool that enriches security events and stores them in Elasticsearch. To manage storage costs and compliance requirements, you need to configure Elasticsearch Index Lifecycle Management (ILM) with LogSentinelAI to automate index rollovers and deletions. This guide walks through the exact implementation found in the `call518/logsentinelai` repository.

## Understanding LogSentinelAI ILM Architecture

LogSentinelAI delegates all index lifecycle decisions to Elasticsearch through a dedicated policy and template. The application code never manages retention logic directly; instead, it writes to a rollover alias and lets Elasticsearch handle the rest.

### Core Components and File Locations

- **[`src/logsentinelai/core/elasticsearch.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/elasticsearch.py)** – Establishes the connection using `ELASTICSEARCH_HOST`, `ELASTICSEARCH_USER`, and `ELASTICSEARCH_PASSWORD`, then writes documents to the index specified by `ELASTICSEARCH_INDEX`.

- **[`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py)** – Defines default connection parameters including the host (`http://localhost:9200`) and the default index prefix `logsentinelai-analysis`.

- **[`INSTALL-and-USAGE.md`](https://github.com/call518/logsentinelai/blob/main/INSTALL-and-USAGE.md)** – Contains the exact `curl` commands required to create the ILM policy and index template.

- **[`Wiki/Home.md`](https://github.com/call518/logsentinelai/blob/main/Wiki/Home.md)** – Documents the default retention phases (hot 7 days, warm 30 days, cold 90 days, delete 365 days) for long-term compliance strategies.

## Creating the ILM Policy for LogSentinelAI

Before running LogSentinelAI, you must create the ILM policy that defines when indices roll over and when they are deleted. The default policy, `logsentinelai-analysis-policy`, is designed for high-volume security logs.

The policy defines two phases:

- **Hot phase** – Rolls over the index when it reaches **10GB** or **1 day**, whichever comes first.
- **Delete phase** – Permanently removes the index after **7 days** from rollover.

Execute this command against your Elasticsearch cluster:

```bash
curl -X PUT "localhost:9200/_ilm/policy/logsentinelai-analysis-policy" \
     -H "Content-Type: application/json" \
     -u elastic:changeme \
     -d '{
       "policy": {
         "phases": {
           "hot": {
             "actions": {
               "rollover": {
                 "max_size": "10gb",
                 "max_age": "1d"
               }
             }
           },
           "delete": {
             "min_age": "7d",
             "actions": { "delete": {} }
           }
         }
       }
     }'

```

## Configuring the Index Template

After creating the policy, you must bind it to the actual indices using an index template. LogSentinelAI uses the template `logsentinelai-analysis-template` to ensure every new index automatically inherits the ILM settings.

The template performs three critical functions:

1. **Pattern matching** – Applies to all indices named `logsentinelai-analysis-*`.
2. **Lifecycle binding** – Sets `index.lifecycle.name` to `logsentinelai-analysis-policy` and defines the rollover alias as `logsentinelai-analysis`.
3. **Mapping configuration** – Defines the schema for enriched log fields, including `geo_point` mappings for GeoIP data.

Run this command to create the template:

```bash
curl -X PUT "localhost:9200/_index_template/logsentinelai-analysis-template" \
     -H "Content-Type: application/json" \
     -u elastic:changeme \
     -d '{
       "index_patterns": ["logsentinelai-analysis-*"],
       "template": {
         "settings": {
           "number_of_shards": 1,
           "number_of_replicas": 1,
           "index.lifecycle.name": "logsentinelai-analysis-policy",
           "index.lifecycle.rollover_alias": "logsentinelai-analysis",
           "index.mapping.total_fields.limit": "10000"
         },
         "mappings": {
           "properties": {
             "timestamp": { "type": "date" },
             "source_ip": { "type": "ip" },
             "geo_location": { "type": "geo_point" }
           }
         }
       }
     }'

```

## Integrating ILM with LogSentinelAI Runtime

Once the policy and template exist, LogSentinelAI automatically handles the rest. The application writes to the rollover alias `logsentinelai-analysis`, and Elasticsearch manages the underlying indices according to your ILM rules.

### Environment Configuration

Override defaults using environment variables defined in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py):

```bash
export ELASTICSEARCH_HOST="https://es-cluster.example.com:9200"
export ELASTICSEARCH_USER="logsentinel"
export ELASTICSEARCH_PASSWORD="secure-password"
export ELASTICSEARCH_INDEX="logsentinelai-analysis"

```

### Running with Elasticsearch Output

Execute the CLI with `--output elasticsearch` to stream enriched logs directly to the managed indices:

```bash
logsentinelai-httpd-access /var/log/apache2/access.log \
    --output elasticsearch \
    --mode realtime

```

Each batch of analyzed logs is indexed into the current write index (e.g., `logsentinelai-analysis-000001`). When the index reaches 10GB or 1 day of age, Elasticsearch automatically rolls over to `logsentinelai-analysis-000002` and begins the lifecycle countdown for the previous index.

## Customizing ILM Retention Policies

The default 7-day retention suits development environments, but production security operations often require longer retention for compliance. You can modify the ILM policy without touching the LogSentinelAI codebase.

To extend retention to 30 days, update the `min_age` in the delete phase:

```bash
curl -X PUT "localhost:9200/_ilm/policy/logsentinelai-analysis-policy" \
     -H "Content-Type: application/json" \
     -u elastic:changeme \
     -d '{
       "policy": {
         "phases": {
           "hot": {
             "actions": {
               "rollover": {
                 "max_size": "10gb",
                 "max_age": "1d"
               }
             }
           },
           "delete": {
             "min_age": "30d",
             "actions": { "delete": {} }
           }
         }
       }
     }'

```

Changes take effect immediately for new rollovers. Existing indices continue with the policy version they were created under unless you manually trigger a retry via the Elasticsearch ILM API.

## Summary

- **LogSentinelAI** delegates index lifecycle management entirely to Elasticsearch ILM, writing to the rollover alias `logsentinelai-analysis` defined in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py).
- You must create the **ILM policy** `logsentinelai-analysis-policy` and **index template** `logsentinelai-analysis-template` via the `curl` commands documented in [`INSTALL-and-USAGE.md`](https://github.com/call518/logsentinelai/blob/main/INSTALL-and-USAGE.md) before running the application.
- The default configuration rolls indices at **10GB or 1 day** and deletes them after **7 days**, but you can customize retention by updating the policy JSON without restarting LogSentinelAI.
- Environment variables (`ELASTICSEARCH_HOST`, `ELASTICSEARCH_USER`, `ELASTICSEARCH_PASSWORD`, `ELASTICSEARCH_INDEX`) allow you to point the CLI at any Elasticsearch cluster without code changes.

## Frequently Asked Questions

### What is the default ILM retention period in LogSentinelAI?

The default ILM policy created for LogSentinelAI retains indices for **7 days** after rollover. According to the Wiki documentation in [`Wiki/Home.md`](https://github.com/call518/logsentinelai/blob/main/Wiki/Home.md), broader organizational strategies may extend this to hot (7 days), warm (30 days), cold (90 days), and delete (365 days) phases, but the initial setup script configures a simple hot-to-delete transition at 7 days.

### How do I change the rollover threshold from 10GB to a different size?

Modify the `max_size` parameter in the hot phase actions of the ILM policy. Send a `PUT` request to `/_ilm/policy/logsentinelai-analysis-policy` with an updated JSON payload containing `"max_size": "50gb"` (or your desired threshold). The change applies to the next rollover; current write indices continue with their existing thresholds.

### Can I use a custom index name instead of logsentinelai-analysis?

Yes. Set the `ELASTICSEARCH_INDEX` environment variable to your preferred alias name before starting LogSentinelAI. You must also update the `index_patterns` in your index template and the `rollover_alias` in both the template and ILM policy to match your custom name. The configuration loader in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py) reads these variables at startup.

### Where does LogSentinelAI store the ILM policy configuration?

LogSentinelAI does not store the ILM policy internally; it resides entirely within your Elasticsearch cluster under the policy name `logsentinelai-analysis-policy`. The application references this policy through the index template setting `index.lifecycle.name` defined in [`INSTALL-and-USAGE.md`](https://github.com/call518/logsentinelai/blob/main/INSTALL-and-USAGE.md). To inspect or modify the policy, use the Elasticsearch REST API at `/_ilm/policy/logsentinelai-analysis-policy`.