How to Debug Elasticsearch Connection and Indexing Issues in LogSentinelAI

Enable debug logging for logsentinelai.elasticsearch and run get_elasticsearch_client().ping() to isolate whether LogSentinelAI fails at the network layer, authentication layer, or indexing layer.

LogSentinelAI streams AI-driven security analysis results to Elasticsearch through the logsentinelai.core.elasticsearch module. When documents fail to appear in your cluster, debugging requires tracing the pipeline from configuration loading in config.py to the final client.index() call. This guide provides a systematic procedure to diagnose connection timeouts, authentication failures, and mapping conflicts in the call518/logsentinelai codebase.

Understanding the Elasticsearch Integration Architecture

The indexing workflow in src/logsentinelai/core/elasticsearch.py follows five distinct stages:

1. Configuration Loading – logsentinelai.core.config reads ELASTICSEARCH_HOST, ELASTICSEARCH_USER, ELASTICSEARCH_PASSWORD, and ELASTICSEARCH_INDEX from the .env file or /etc/logsentinelai.config (config.py#L45-L48).

2. Client Creation – get_elasticsearch_client() instantiates the Elasticsearch object and verifies connectivity via client.ping() (elasticsearch.py#L19-L35).

3. Document Enrichment – The payload is augmented with host metadata and @timestamp fields using get_host_metadata() from logsentinelai.utils.general (elasticsearch.py#L70-L77).

4. Indexing – client.index(index=ELASTICSEARCH_INDEX, id=doc_id, document=enriched_data) transmits the JSON document (elasticsearch.py#L20-L33).

5. Result Handling – The function returns True when result is 'created' or 'updated'; otherwise, it logs detailed error information (elasticsearch.py#L27-L36).

Because each stage uses the dedicated logsentinelai.elasticsearch logger, you can pinpoint failures by inspecting log output.

Step-by-Step Debugging Procedure

Verify Configuration Loading

Start by confirming that environment variables are parsed correctly. Misconfigured hosts or missing credentials are the most common root causes.

from logsentinelai.core import config

print("Host:", config.ELASTICSEARCH_HOST)
print("User:", config.ELASTICSEARCH_USER)
print("Index:", config.ELASTICSEARCH_INDEX)

If these values are None or incorrect, check your .env file or /etc/logsentinelai.config permissions.

Test Client Connectivity and Ping

Once configuration is valid, verify that the Python client can reach the cluster. The get_elasticsearch_client() function in elasticsearch.py automatically calls client.ping().

from logsentinelai.core.elasticsearch import get_elasticsearch_client

client = get_elasticsearch_client()
if client:
    print("Ping successful:", client.ping())
else:
    print("Client creation failed")

A False or None result indicates network issues, TLS misconfiguration, or authentication failures.

Inspect Connection and Authentication Errors

Enable debug logging to capture the exact exception type raised by the Elasticsearch library. The logsentinelai.elasticsearch logger records ConnectionError and RequestError details.

import logging

logging.getLogger("logsentinelai.elasticsearch").setLevel(logging.DEBUG)
logging.getLogger("elasticsearch").setLevel(logging.DEBUG)

Re-run your indexing operation and watch for:

• ConnectionError – Host unreachable, DNS failure, or firewall blocking port 9200.
• AuthenticationException (HTTP 401) – Invalid ELASTICSEARCH_USER or ELASTICSEARCH_PASSWORD.
• SSLError – Certificate verification failed when verify_certs is enabled.

Validate Index Existence and Mapping

If the client connects but indexing fails, verify that ELASTICSEARCH_INDEX exists and accepts the document schema.

curl -s -u "$ELASTICSEARCH_USER:$ELASTICSEARCH_PASSWORD" \
  "$ELASTICSEARCH_HOST/$ELASTICSEARCH_INDEX?pretty"

If the index is missing, create it manually or ensure the Elasticsearch user has the create_index privilege. Check for mapper_parsing_exception errors that indicate field type conflicts.

Isolate Indexing with Manual Test

Bypass the full LogSentinelAI pipeline to test raw indexing capabilities. This determines whether the issue lies in the Elasticsearch layer or the data preparation layer.

from logsentinelai.core.elasticsearch import get_elasticsearch_client
from logsentinelai.core import config
import datetime

client = get_elasticsearch_client()
if not client:
    raise SystemExit("Client unavailable")

doc = {
    "message": "LogSentinelAI debug document",
    "@timestamp": datetime.datetime.utcnow().isoformat(),
    "host": "debug-test"
}

response = client.index(
    index=config.ELASTICSEARCH_INDEX,
    id="debug_001",
    document=doc
)

print("Index result:", response.get("result"))

A successful created or updated result confirms that credentials, network, and index mapping are correct.

Check Server-Side Elasticsearch Logs

When client-side debugging is inconclusive, inspect Elasticsearch server logs for authentication failures, shard rejections, or circuit breaker exceptions.

# Systemd-based installations

journalctl -u elasticsearch -f

# Or tail the log directory

tail -f /var/log/elasticsearch/*.log

Look for action [indices:data/write/index] denials that indicate insufficient privileges for the LogSentinelAI user.

Common Failure Scenarios and Remedies

Symptom	Likely Cause	Fix
Elasticsearch connection error and client is None	Wrong ELASTICSEARCH_HOST URL, firewall rule, or Elasticsearch service not running.	Verify the URL includes protocol and port (https://localhost:9200). Test with curl.
HTTP 401 Unauthorized	Invalid ELASTICSEARCH_USER or ELASTICSEARCH_PASSWORD.	Check credentials against the Elasticsearch security API (GET /_security/user).
index_not_found_exception	ELASTICSEARCH_INDEX does not exist.	Create the index manually or grant create_index privilege to the LogSentinelAI role.
mapper_parsing_exception	Document fields conflict with existing index mapping (e.g., sending string to integer field).	Review mapping with GET /index/_mapping and align payload structure in elasticsearch.py.
TLS handshake failure	verify_certs=False suppresses errors but server requires valid CA.	Set verify_certs=True and provide ca_certs path in get_elasticsearch_client() arguments.
Large document rejection	Payload exceeds http.max_content_length (default 100MB).	Increase limit in elasticsearch.yml or reduce batch size in the indexing logic.

Practical Debugging Scripts

Minimal Health Check Script

Run this standalone script to verify connectivity without executing the full LogSentinelAI pipeline.

# health_check_es.py

import logging
from logsentinelai.core.elasticsearch import get_elasticsearch_client
from logsentinelai.core import config

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("logsentinelai.elasticsearch").setLevel(logging.DEBUG)

def main():
    print("Configured host :", config.ELASTICSEARCH_HOST)
    print("Configured index:", config.ELASTICSEARCH_INDEX)

    client = get_elasticsearch_client()
    if not client:
        print("❌ Could not create Elasticsearch client")
        return

    print("🔎 Ping result :", client.ping())
    health = client.cluster.health()
    print("🩺 Cluster health :", health)

    if client.indices.exists(index=config.ELASTICSEARCH_INDEX):
        print(f"✅ Index '{config.ELASTICSEARCH_INDEX}' is present")
    else:
        print(f"⚠️ Index '{config.ELASTICSEARCH_INDEX}' not found")

if __name__ == "__main__":
    main()

Execute with:

python health_check_es.py

Isolated Indexing Test

Use this snippet to test document insertion without triggering the analysis pipeline.

# test_index.py

from logsentinelai.core.elasticsearch import get_elasticsearch_client
from logsentinelai.core import config
import json
import datetime

client = get_elasticsearch_client()
if not client:
    raise SystemExit("Unable to create Elasticsearch client")

doc_id = f"debug_{datetime.datetime.utcnow().isoformat()}"
payload = {
    "message": "debug test from LogSentinelAI",
    "@timestamp": datetime.datetime.utcnow().isoformat(),
    "host": "debug-host"
}

response = client.index(
    index=config.ELASTICSEARCH_INDEX,
    id=doc_id,
    document=doc,
)

print("Response:", json.dumps(response, indent=2, ensure_ascii=False))

Enabling Verbose Application Logging

Add this configuration to capture internal library chatter.

import logging
from logsentinelai.core.commons import setup_logger

logger = setup_logger("logsentinelai.elasticsearch")
logger.setLevel(logging.DEBUG)
logging.getLogger("elasticsearch").setLevel(logging.DEBUG)

Alternatively, set the environment variable before launching:

export LOG_LEVEL=DEBUG
python -m logsentinelai.cli

Key Source Files Reference

File	Role	Link
src/logsentinelai/core/elasticsearch.py	Core integration: client creation, payload enrichment, indexing logic, and error handling.	elasticsearch.py
src/logsentinelai/core/config.py	Loads environment variables including ELASTICSEARCH_HOST, ELASTICSEARCH_USER, and ELASTICSEARCH_INDEX.	config.py
src/logsentinelai/core/commons.py	Provides setup_logger used by the Elasticsearch module for structured debug output.	commons.py
src/logsentinelai/utils/general.py	Supplies get_host_metadata() that enriches documents with host information before indexing.	general.py
src/logsentinelai/cli.py	Entry point that orchestrates the analysis pipeline and triggers Elasticsearch indexing.	cli.py

Summary

• Configuration errors are the most common root cause; verify ELASTICSEARCH_HOST, ELASTICSEARCH_USER, and ELASTICSEARCH_INDEX via logsentinelai.core.config.
• Connectivity issues surface in get_elasticsearch_client(); use client.ping() and the ConnectionError exception handler in elasticsearch.py to diagnose network or TLS problems.
• Authentication failures return HTTP 401; confirm credentials in your .env file match Elasticsearch security settings.
• Indexing errors such as index_not_found_exception or mapper_parsing_exception require checking index existence and mapping compatibility using the isolated test scripts provided.
• Debug logging via logsentinelai.elasticsearch and the elasticsearch library logger provides the exact exception messages and request traces needed for resolution.

Frequently Asked Questions

How do I enable debug logging for Elasticsearch operations in LogSentinelAI?

Set the logger level to DEBUG for both the application and the underlying Elasticsearch library. Import setup_logger from logsentinelai.core.commons and call setup_logger("logsentinelai.elasticsearch").setLevel(logging.DEBUG). Alternatively, set the environment variable LOG_LEVEL=DEBUG before launching the CLI. This exposes the exact HTTP requests, ping results, and index responses in logsentinelai.log.

What does the "index_not_found_exception" error mean in LogSentinelAI?

This error indicates that the index specified in ELASTICSEARCH_INDEX does not exist on the cluster. LogSentinelAI does not automatically create indices with custom mappings. Resolve this by manually creating the index via the Elasticsearch API or granting the create_index privilege to the LogSentinelAI user so the first document insertion auto-creates the index.

Why is LogSentinelAI failing with a 401 Unauthorized error when connecting to Elasticsearch?

HTTP 401 errors occur when ELASTICSEARCH_USER or ELASTICSEARCH_PASSWORD in your .env file do not match valid credentials in the Elasticsearch cluster. Verify the credentials using curl -u user:pass $ELASTICSEARCH_HOST. If using API keys instead of basic auth, ensure the ELASTICSEARCH_USER is set to the API key ID and ELASTICSEARCH_PASSWORD to the API key secret.

How can I test if LogSentinelAI can reach my Elasticsearch cluster without running a full analysis?

Use the minimal health-check script that imports get_elasticsearch_client from logsentinelai.core.elasticsearch and calls client.ping(). This validates that the configuration loads correctly, the network path is open, and authentication succeeds without triggering the full AI analysis pipeline. If ping() returns True, the cluster is reachable and ready to receive documents.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}