# How to Migrate from Traditional Regex-Based Log Analysis to LogSentinelAI

> Transition from regex log analysis to LogSentinelAI. Learn how to automate structured security intelligence extraction from raw logs using LLMs and Pydantic.

- Repository: [JungJungIn/logsentinelai](https://github.com/call518/logsentinelai)
- Tags: migration-guide
- Published: 2026-02-26

---

**LogSentinelAI replaces brittle regular-expression parsers with a declarative, LLM-driven pipeline that automatically extracts structured security intelligence from raw logs using Pydantic schemas and configurable prompts.**

Migrating from traditional regex-based log analysis to LogSentinelAI involves shifting from pattern-matching code to schema-first declarative configuration. The `call518/logsentinelai` repository provides a modular architecture where you define the output structure once, and the framework handles chunking, LLM invocation, validation, and delivery. This guide maps the exact source code locations and functions you need to complete the migration in four logical steps.

## The Four-Step Migration Process

The migration path follows the repository’s core abstraction layers, allowing you to swap regex logic for LLM-powered extraction without rewriting ingestion or storage code.

### Step 1: Define the Desired Output with Pydantic Models

Replace your regex capture groups with **Pydantic models** that declare the fields you want extracted. Each model becomes the schema the LLM populates.

In [`src/logsentinelai/analyzers/linux_system.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/analyzers/linux_system.py) (lines 41‑68), the `SecurityEvent`, `Statistics`, and `LogAnalysis` models demonstrate the canonical pattern:

- Use `Field(description=...)` to guide the LLM’s extraction logic
- Define enums for constrained values like severity levels
- Mark optional fields with `Optional[Type]` for flexible parsing

This declarative approach eliminates the need to maintain complex regex patterns for every log variant.

### Step 2: Create or Adapt LLM Prompts

Create a prompt template that instructs the LLM how to map raw log lines to your Pydantic schema. The repository ships ready-made prompts for built-in analyzers that you can copy and customize.

The prompt generation pattern is implemented in [`src/logsentinelai/core/prompts.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/prompts.py). Functions like `get_linux_system_prompt()` return templates that include the `{model_schema}` placeholder, which the framework dynamically populates with your Pydantic model’s JSON schema.

Your prompt should:
1. Reference the schema placeholder explicitly
2. Define how to handle multi-line log entries
3. Specify default values for ambiguous fields

### Step 3: Wire the Analysis Pipeline

Invoke the generic runner functions that orchestrate file discovery, chunking, LLM calls, and validation. You pass the *log type*, *schema class*, and *prompt*; the framework handles the rest.

In [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py), two critical functions provide this orchestration:

- **`run_generic_batch_analysis()`** (lines 53‑120): Processes historical log files in configurable chunks
- **`run_generic_realtime_analysis()`** (lines 518‑590): Monitors live log streams with the same validation and enrichment pipeline

Both functions automatically handle:
- File pattern expansion and rotation detection
- GeoIP enrichment via `enrich_source_ips_with_geoip`
- Elasticsearch indexing via `send_to_elasticsearch`
- Telegram alerting for critical severity events

### Step 4: Configure Environment and Providers

Set the LLM provider, model selection, API endpoints, and integration targets via environment variables or a configuration file. The configuration loader normalizes these values across the entire stack.

In [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py), the `_load_values()` function (lines 55‑124) reads from `/etc/logsentinelai.config` or `./.env`, while `apply_config()` (lines 134‑188) exposes module-level globals including:

- `LLM_PROVIDER` (ollama, vllm, openai, gemini)
- `LOG_PATHS` (glob patterns for log discovery)
- `ELASTICSEARCH_HOST` and `ELASTICSEARCH_INDEX`
- `TELEGRAM_ENABLED` and `TELEGRAM_BOT_TOKEN`

## Architectural Flow and Core Components

Understanding the internal data flow helps debug migration issues and optimize chunk sizes.

### Configuration Layer

At import time, `apply_config()` in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py) loads system-wide or local environment files. These values control provider selection and output destinations.

### Logger and Context Injection

The `setup_logger()` function in [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py) (lines 24‑49) creates a contextual logger that injects the current `log_type` into every record. This provides traceability through the chunk processing pipeline.

### LLM Abstraction

[`src/logsentinelai/core/llm.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/llm.py) provides provider-agnostic interfaces:

- **`initialize_llm_model()`** (lines 30‑71): Builds clients for Ollama, vLLM, OpenAI, or Gemini based on `LLM_PROVIDER`
- **`generate_with_model()`** (lines 77‑144): Sends the prompt plus schema (for non-Gemini providers) and returns structured JSON

### Chunk Processing and Validation

The `_process_file_streaming_batch()` function in [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py) (lines 172‑236) streams log files while respecting snapshot boundaries and detecting rotation. Each chunk passes through `process_log_chunk()` (lines 26‑44), which:

1. Adds metadata (timestamps, token counts, access mode)
2. Validates the LLM response against your Pydantic schema
3. Triggers GeoIP enrichment and Elasticsearch delivery

### Real-Time Monitoring

The `RealtimeLogMonitor` class watches files for new entries, emitting line groups that feed through the same `process_log_chunk()` path used in batch mode, ensuring consistent validation and alerting.

### Output and Alerting

Structured JSON is indexed into Elasticsearch via [`core/elasticsearch.py`](https://github.com/call518/logsentinelai/blob/main/core/elasticsearch.py). Critical events trigger Telegram notifications through [`src/logsentinelai/utils/telegram_alert.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/utils/telegram_alert.py) when `TELEGRAM_ENABLED` is set.

## Practical Migration Examples

### Defining a Custom Schema

Create a Pydantic model for your proprietary log format:

```python
from pydantic import BaseModel, Field
from enum import Enum
from typing import Optional, List

class EventSeverity(str, Enum):
    CRITICAL = "CRITICAL"
    HIGH = "HIGH"
    MEDIUM = "MEDIUM"
    LOW = "LOW"
    INFO = "INFO"

class CustomEvent(BaseModel):
    event_type: str = Field(description="Type of the event")
    severity: EventSeverity
    message: str = Field(description="Original log line")
    timestamp: str = Field(description="ISO-8601 timestamp")
    source_ip: Optional[str] = Field(description="IP address, if any")
    tags: List[str] = Field(default_factory=list, description="User-defined tags")

```

Reference the `SecurityEvent` definition in [`src/logsentinelai/analyzers/linux_system.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/analyzers/linux_system.py) (lines 41‑53) for additional field patterns.

### Writing the Analysis Prompt

Define a prompt function that references your schema:

```python
def get_custom_prompt():
    return """You are a security analyst.  
Extract events from the following log lines and output JSON matching this schema:

{model_schema}

Each log line is given after a line that starts with "LOG:".  
Use the most specific `event_type` you can infer.  
If the line contains an IP address, put it in `source_ip`.  
If you cannot determine severity, use INFO.

LOG:
{logs}
"""

```

This follows the pattern in [`src/logsentinelai/core/prompts.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/prompts.py) where `get_linux_system_prompt()` constructs similar templates.

### Running Batch Analysis via Python

Execute the migration programmatically:

```python
from logsentinelai.core.commons import run_generic_batch_analysis
from logsentinelai.core.prompts import get_custom_prompt

run_generic_batch_analysis(
    log_type="custom_log",
    analysis_schema_class=CustomEvent,
    prompt_template=get_custom_prompt(),
    analysis_title="Custom Log Analysis",
    log_path="samples/custom.log",
    chunk_size=20,
    remote_mode="local"
)

```

The `run_generic_batch_analysis()` function in [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py) (lines 53‑120) manages the entire workflow from file reading to Elasticsearch indexing.

### CLI Execution

Run migrated analyzers without code changes:

```bash

# Batch processing

logsentinelai-custom-log --log-path "samples/*.log"

# Real-time monitoring

logsentinelai-custom-log --mode realtime --log-path /var/log/custom.log

```

CLI entry points are wired in [`src/logsentinelai/cli.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/cli.py) and analyzer-specific scripts.

## Summary

- **Schema-first migration**: Replace regex capture groups with Pydantic models in `src/logsentinelai/analyzers/`
- **Prompt-driven extraction**: Use [`src/logsentinelai/core/prompts.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/prompts.py) templates to guide LLM output without parsing logic
- **Unified runners**: `run_generic_batch_analysis()` and `run_generic_realtime_analysis()` in [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py) handle all orchestration
- **Environment configuration**: Set providers and targets in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py) via `.env` or system config files
- **Drop-in replacement**: The framework’s chunking, validation, GeoIP enrichment, and alerting layers work identically across all log types

## Frequently Asked Questions

### Do I need to remove my existing regex parsers before migrating?

No. LogSentinelAI operates alongside existing infrastructure. You can run both systems in parallel during validation, then decommission regex parsers once the LLM extraction accuracy meets your requirements. The framework reads raw log files directly without modifying your current ingestion pipeline.

### What LLM providers does LogSentinelAI support?

The `initialize_llm_model()` function in [`src/logsentinelai/core/llm.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/llm.py) (lines 30‑71) supports Ollama, vLLM, OpenAI, and Gemini. You switch providers by changing the `LLM_PROVIDER` environment variable in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py) without modifying analysis code. Each provider route handles authentication and request formatting internally.

### How does LogSentinelAI handle large log files?

The `_process_file_streaming_batch()` function in [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py) (lines 172‑236) processes files in configurable chunks (default 20 lines) to respect LLM context windows and memory constraints. It detects file rotation and respects snapshot boundaries, ensuring no data loss during high-volume ingestion. Adjust `CHUNK_SIZE` in your configuration to optimize throughput versus token cost.

### Can I use LogSentinelAI for real-time log monitoring?

Yes. The `run_generic_realtime_analysis()` function in [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py) (lines 518‑590) provides the same schema validation and enrichment as batch mode but operates on live file streams. The `RealtimeLogMonitor` class watches files for new entries and processes them through identical validation and alerting pipelines, enabling immediate Telegram notifications for critical security events.