How to Integrate LogSentinelAI with Elasticsearch for SIEM

LogSentinelAI integrates with Elasticsearch by enriching LLM-analyzed log chunks with host metadata and indexing them into configurable indices via the send_to_elasticsearch_raw() function, enabling SIEM visualization through Kibana while supporting real-time Telegram alerts for high-severity events.

LogSentinelAI provides a modular pipeline for LLM-based log analysis that natively supports Elasticsearch integration for SIEM workflows. By configuring environment variables in the call518/logsentinelai repository and utilizing the built-in enrichment engine, security teams can automatically index structured threat intelligence into Elasticsearch indices for real-time dashboarding and alerting.

Elasticsearch Configuration Environment

LogSentinelAI reads Elasticsearch connection parameters from environment variables defined in src/logsentinelai/core/config.py. The system exposes module globals including ELASTICSEARCH_HOST, ELASTICSEARCH_USER, ELASTICSEARCH_PASSWORD, and ELASTICSEARCH_INDEX that control where analyzed logs are stored.

Create a .env file or export variables directly:

ELASTICSEARCH_HOST=http://localhost:9200
ELASTICSEARCH_USER=elastic
ELASTICSEARCH_PASSWORD=changeme
ELASTICSEARCH_INDEX=logsentinelai-analysis

These values are loaded when config.apply_config() executes during package initialization. The configuration module validates that these globals are available before any indexing operations begin.

Core Integration Architecture

Elasticsearch Client Initialization

The src/logsentinelai/core/elasticsearch.py file manages the Elasticsearch connection through get_elasticsearch_client(). This function instantiates a reusable Elasticsearch object and verifies connectivity before returning the client to the calling code.

Data Enrichment Pipeline

Before indexing, the send_to_elasticsearch_raw() function performs critical enrichment steps defined in src/logsentinelai/core/elasticsearch.py. The pipeline:

• Generates unique document IDs using the format log_type_YYYYMMDD_hhmmss_ffffff[_chunk_N]
• Attaches host-level metadata via get_host_metadata()
• Adds precise timestamps and GeoIP data when available
• Validates event severity levels for alert triggers

Centralized Routing

The src/logsentinelai/core/commons.py file provides a higher-level abstraction that forwards enriched data from various analyzers to the Elasticsearch indexing function. This ensures consistent data formatting regardless of the log source type.

End-to-End Data Flow

The integration follows a five-stage pipeline orchestrated by src/logsentinelai/cli.py:

1. Log Ingestion – An analyzer (such as analyzers/httpd_access.py) reads raw log chunks from the filesystem
2. LLM Analysis – The system sends chunks to the LLM via core/llm.py and receives structured JSON payloads identifying security events
3. Enrichment – send_to_elasticsearch_raw() attaches host metadata, timestamps, and document identifiers
4. Alert Evaluation – If TELEGRAM_ENABLED is true and events meet the TELEGRAM_ALERT_LEVEL threshold, the system dispatches real-time notifications
5. Indexing – The final JSON document is pushed to the configured Elasticsearch index via client.index(), returning True only when the response contains "result": "created" or "updated"

Implementation Methods

Method 1: CLI-Based Integration

The simplest approach uses the built-in command-line interface to process logs and automatically index results:

pip install -r requirements.txt
logsentinelai run --log-type httpd_access --log-path /var/log/apache/access.log

The CLI invokes process_log_chunk() from src/logsentinelai/core/commons.py, which handles the complete enrichment and indexing workflow without requiring custom code.

Method 2: Programmatic Python Integration

For custom SIEM pipelines, import the Elasticsearch functions directly:

from logsentinelai.core.elasticsearch import send_to_elasticsearch_raw

analysis_result = {
    "events": [
        {"severity": "CRITICAL", "event_type": "SQLi", "description": "Possible SQL injection detected"},
        {"severity": "HIGH", "event_type": "PortScan", "description": "Multiple ports probed"}
    ],
    "summary": "2 security events detected"
}

success = send_to_elasticsearch_raw(
    data=analysis_result,
    log_type="httpd_access",
    chunk_id=1
)

print(f"Document indexed: {success}")

This approach allows integration with existing Python-based security orchestration tools while maintaining LogSentinelAI's enrichment capabilities.

Method 3: Custom Analyzer Integration

When building custom log analyzers, leverage the commons helper to ensure consistent Elasticsearch formatting:

from logsentinelai.core.commons import process_log_chunk

# Custom analyzer implementation

def my_custom_analyzer(log_path):
    # Analysis logic here

    results = analyze_logs(log_path)
    return results

# The commons module handles Elasticsearch routing automatically

Enabling Real-Time Security Alerts

LogSentinelAI supports Telegram integration for immediate notification of high-severity events. Configure these additional environment variables alongside your Elasticsearch settings:

TELEGRAM_ENABLED=true
TELEGRAM_TOKEN=YOUR_BOT_TOKEN
TELEGRAM_CHAT_ID=YOUR_CHAT_ID
TELEGRAM_ALERT_LEVEL=HIGH

When send_to_elasticsearch_raw() processes events with severity levels matching or exceeding TELEGRAM_ALERT_LEVEL, it triggers utils.telegram_alert.send_telegram_alert() to dispatch formatted alerts containing event details and host metadata before completing the Elasticsearch indexing operation.

Verifying the Integration

After running your first analysis, verify the integration by querying the target index:

curl -X GET "localhost:9200/logsentinelai-analysis/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{"query": {"match_all": {}}}'

Successful integration returns documents containing the enriched fields added by send_to_elasticsearch_raw(), including host_metadata, timestamp, and the unique chunk_id identifier.

Summary

• Configuration: Set ELASTICSEARCH_HOST, ELASTICSEARCH_USER, ELASTICSEARCH_PASSWORD, and ELASTICSEARCH_INDEX environment variables in src/logsentinelai/core/config.py before running analyses
• Client Management: The get_elasticsearch_client() function in src/logsentinelai/core/elasticsearch.py handles connection pooling and health checks
• Enrichment: send_to_elasticsearch_raw() automatically adds host metadata, timestamps, and unique document IDs using the format log_type_YYYYMMDD_hhmmss_ffffff[_chunk_N]
• Routing: src/logsentinelai/core/commons.py provides the central routing layer that connects analyzers to Elasticsearch storage
• Alerts: Optional Telegram notifications trigger when event severity meets the configured TELEGRAM_ALERT_LEVEL threshold

Frequently Asked Questions

How do I configure Elasticsearch authentication credentials?

LogSentinelAI uses environment variables defined in src/logsentinelai/core/config.py. Export ELASTICSEARCH_USER and ELASTICSEARCH_PASSWORD before starting the application, or define them in a .env file. The get_elasticsearch_client() function reads these globals to instantiate the authenticated client connection.

What happens if Elasticsearch is unreachable during analysis?

The send_to_elasticsearch_raw() function returns True only when the Elasticsearch response contains "result": "created" or "updated". If the connection fails or the cluster returns an error, the function returns False, allowing calling code in src/logsentinelai/core/commons.py to handle retries or logging failures appropriately.

Can I customize the index name or document ID format?

Yes. The ELASTICSEARCH_INDEX environment variable controls the target index name. While the document ID format log_type_YYYYMMDD_hhmmss_ffffff[_chunk_N] is generated automatically within send_to_elasticsearch_raw(), you can modify the indexing logic in src/logsentinelai/core/elasticsearch.py if your SIEM requires alternative naming conventions.

Is Telegram alerting required for Elasticsearch integration?

No. Telegram integration is optional. If TELEGRAM_ENABLED is not set to true, the system skips alert generation and proceeds directly to indexing. The alerting logic is embedded in send_to_elasticsearch_raw() but only executes when the configuration variables are present and event severity meets the TELEGRAM_ALERT_LEVEL threshold.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}