How to Migrate from Traditional Regex-Based Log Analysis to LogSentinelAI

LogSentinelAI replaces brittle regular-expression parsers with a declarative, LLM-driven pipeline that automatically extracts structured security intelligence from raw logs using Pydantic schemas and configurable prompts.

Migrating from traditional regex-based log analysis to LogSentinelAI involves shifting from pattern-matching code to schema-first declarative configuration. The call518/logsentinelai repository provides a modular architecture where you define the output structure once, and the framework handles chunking, LLM invocation, validation, and delivery. This guide maps the exact source code locations and functions you need to complete the migration in four logical steps.

The Four-Step Migration Process

The migration path follows the repository’s core abstraction layers, allowing you to swap regex logic for LLM-powered extraction without rewriting ingestion or storage code.

Step 1: Define the Desired Output with Pydantic Models

Replace your regex capture groups with Pydantic models that declare the fields you want extracted. Each model becomes the schema the LLM populates.

In src/logsentinelai/analyzers/linux_system.py (lines 41‑68), the SecurityEvent, Statistics, and LogAnalysis models demonstrate the canonical pattern:

• Use Field(description=...) to guide the LLM’s extraction logic
• Define enums for constrained values like severity levels
• Mark optional fields with Optional[Type] for flexible parsing

This declarative approach eliminates the need to maintain complex regex patterns for every log variant.

Step 2: Create or Adapt LLM Prompts

Create a prompt template that instructs the LLM how to map raw log lines to your Pydantic schema. The repository ships ready-made prompts for built-in analyzers that you can copy and customize.

The prompt generation pattern is implemented in src/logsentinelai/core/prompts.py. Functions like get_linux_system_prompt() return templates that include the {model_schema} placeholder, which the framework dynamically populates with your Pydantic model’s JSON schema.

Your prompt should:

1. Reference the schema placeholder explicitly
2. Define how to handle multi-line log entries
3. Specify default values for ambiguous fields

Step 3: Wire the Analysis Pipeline

Invoke the generic runner functions that orchestrate file discovery, chunking, LLM calls, and validation. You pass the log type, schema class, and prompt; the framework handles the rest.

In src/logsentinelai/core/commons.py, two critical functions provide this orchestration:

• run_generic_batch_analysis() (lines 53‑120): Processes historical log files in configurable chunks
• run_generic_realtime_analysis() (lines 518‑590): Monitors live log streams with the same validation and enrichment pipeline

Both functions automatically handle:

• File pattern expansion and rotation detection
• GeoIP enrichment via enrich_source_ips_with_geoip
• Elasticsearch indexing via send_to_elasticsearch
• Telegram alerting for critical severity events

Step 4: Configure Environment and Providers

Set the LLM provider, model selection, API endpoints, and integration targets via environment variables or a configuration file. The configuration loader normalizes these values across the entire stack.

In src/logsentinelai/core/config.py, the _load_values() function (lines 55‑124) reads from /etc/logsentinelai.config or ./.env, while apply_config() (lines 134‑188) exposes module-level globals including:

• LLM_PROVIDER (ollama, vllm, openai, gemini)
• LOG_PATHS (glob patterns for log discovery)
• ELASTICSEARCH_HOST and ELASTICSEARCH_INDEX
• TELEGRAM_ENABLED and TELEGRAM_BOT_TOKEN

Architectural Flow and Core Components

Understanding the internal data flow helps debug migration issues and optimize chunk sizes.

Configuration Layer

At import time, apply_config() in src/logsentinelai/core/config.py loads system-wide or local environment files. These values control provider selection and output destinations.

Logger and Context Injection

The setup_logger() function in src/logsentinelai/core/commons.py (lines 24‑49) creates a contextual logger that injects the current log_type into every record. This provides traceability through the chunk processing pipeline.

LLM Abstraction

src/logsentinelai/core/llm.py provides provider-agnostic interfaces:

• initialize_llm_model() (lines 30‑71): Builds clients for Ollama, vLLM, OpenAI, or Gemini based on LLM_PROVIDER
• generate_with_model() (lines 77‑144): Sends the prompt plus schema (for non-Gemini providers) and returns structured JSON

Chunk Processing and Validation

The _process_file_streaming_batch() function in src/logsentinelai/core/commons.py (lines 172‑236) streams log files while respecting snapshot boundaries and detecting rotation. Each chunk passes through process_log_chunk() (lines 26‑44), which:

1. Adds metadata (timestamps, token counts, access mode)
2. Validates the LLM response against your Pydantic schema
3. Triggers GeoIP enrichment and Elasticsearch delivery

Real-Time Monitoring

The RealtimeLogMonitor class watches files for new entries, emitting line groups that feed through the same process_log_chunk() path used in batch mode, ensuring consistent validation and alerting.

Output and Alerting

Structured JSON is indexed into Elasticsearch via core/elasticsearch.py. Critical events trigger Telegram notifications through src/logsentinelai/utils/telegram_alert.py when TELEGRAM_ENABLED is set.

Practical Migration Examples

Defining a Custom Schema

Create a Pydantic model for your proprietary log format:

from pydantic import BaseModel, Field
from enum import Enum
from typing import Optional, List

class EventSeverity(str, Enum):
    CRITICAL = "CRITICAL"
    HIGH = "HIGH"
    MEDIUM = "MEDIUM"
    LOW = "LOW"
    INFO = "INFO"

class CustomEvent(BaseModel):
    event_type: str = Field(description="Type of the event")
    severity: EventSeverity
    message: str = Field(description="Original log line")
    timestamp: str = Field(description="ISO-8601 timestamp")
    source_ip: Optional[str] = Field(description="IP address, if any")
    tags: List[str] = Field(default_factory=list, description="User-defined tags")

Reference the SecurityEvent definition in src/logsentinelai/analyzers/linux_system.py (lines 41‑53) for additional field patterns.

Writing the Analysis Prompt

Define a prompt function that references your schema:

def get_custom_prompt():
    return """You are a security analyst.  
Extract events from the following log lines and output JSON matching this schema:

{model_schema}

Each log line is given after a line that starts with "LOG:".  
Use the most specific `event_type` you can infer.  
If the line contains an IP address, put it in `source_ip`.  
If you cannot determine severity, use INFO.

LOG:
{logs}
"""

This follows the pattern in src/logsentinelai/core/prompts.py where get_linux_system_prompt() constructs similar templates.

Running Batch Analysis via Python

Execute the migration programmatically:

from logsentinelai.core.commons import run_generic_batch_analysis
from logsentinelai.core.prompts import get_custom_prompt

run_generic_batch_analysis(
    log_type="custom_log",
    analysis_schema_class=CustomEvent,
    prompt_template=get_custom_prompt(),
    analysis_title="Custom Log Analysis",
    log_path="samples/custom.log",
    chunk_size=20,
    remote_mode="local"
)

The run_generic_batch_analysis() function in src/logsentinelai/core/commons.py (lines 53‑120) manages the entire workflow from file reading to Elasticsearch indexing.

CLI Execution

Run migrated analyzers without code changes:

# Batch processing

logsentinelai-custom-log --log-path "samples/*.log"

# Real-time monitoring

logsentinelai-custom-log --mode realtime --log-path /var/log/custom.log

CLI entry points are wired in src/logsentinelai/cli.py and analyzer-specific scripts.

Summary

• Schema-first migration: Replace regex capture groups with Pydantic models in src/logsentinelai/analyzers/
• Prompt-driven extraction: Use src/logsentinelai/core/prompts.py templates to guide LLM output without parsing logic
• Unified runners: run_generic_batch_analysis() and run_generic_realtime_analysis() in src/logsentinelai/core/commons.py handle all orchestration
• Environment configuration: Set providers and targets in src/logsentinelai/core/config.py via .env or system config files
• Drop-in replacement: The framework’s chunking, validation, GeoIP enrichment, and alerting layers work identically across all log types

Frequently Asked Questions

Do I need to remove my existing regex parsers before migrating?

No. LogSentinelAI operates alongside existing infrastructure. You can run both systems in parallel during validation, then decommission regex parsers once the LLM extraction accuracy meets your requirements. The framework reads raw log files directly without modifying your current ingestion pipeline.

What LLM providers does LogSentinelAI support?

The initialize_llm_model() function in src/logsentinelai/core/llm.py (lines 30‑71) supports Ollama, vLLM, OpenAI, and Gemini. You switch providers by changing the LLM_PROVIDER environment variable in src/logsentinelai/core/config.py without modifying analysis code. Each provider route handles authentication and request formatting internally.

How does LogSentinelAI handle large log files?

The _process_file_streaming_batch() function in src/logsentinelai/core/commons.py (lines 172‑236) processes files in configurable chunks (default 20 lines) to respect LLM context windows and memory constraints. It detects file rotation and respects snapshot boundaries, ensuring no data loss during high-volume ingestion. Adjust CHUNK_SIZE in your configuration to optimize throughput versus token cost.

Can I use LogSentinelAI for real-time log monitoring?

Yes. The run_generic_realtime_analysis() function in src/logsentinelai/core/commons.py (lines 518‑590) provides the same schema validation and enrichment as batch mode but operates on live file streams. The RealtimeLogMonitor class watches files for new entries and processes them through identical validation and alerting pipelines, enabling immediate Telegram notifications for critical security events.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}