# Understanding the Difference Between Batch and Realtime Analysis Modes in LogSentinelAI

> Discover the difference between batch and realtime analysis modes in LogSentinelAI. Learn when to use static file processing versus infinite monitoring for your logs.

- Repository: [JungJungIn/logsentinelai](https://github.com/call518/logsentinelai)
- Tags: deep-dive
- Published: 2026-02-26

---

**Batch mode processes static log files in finite, chunked streams via `run_generic_batch_analysis()`, while realtime mode runs an infinite monitoring loop through `run_generic_realtime_analysis()` and `RealtimeLogMonitor` to analyze new lines as they are written.**

The LogSentinelAI open-source repository provides dual execution paths for log analysis, allowing security teams to choose between **batch** and **realtime** analysis modes depending on operational requirements. Understanding the difference between batch and realtime analysis modes in LogSentinelAI is essential for selecting the correct pipeline for forensic investigations versus continuous security monitoring.

## Core Architectural Differences

### Batch Mode Implementation

In [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py), the `run_generic_batch_analysis()` function (lines 53-58) serves as the entry point for batch processing. This mode expands wildcard patterns like `/var/log/apache*.log` into discrete file lists, then iterates through each file using `_process_file_streaming_batch()`. The function streams content in configurable chunks until all files are exhausted, then prints a summary and terminates.

### Realtime Mode Implementation

Realtime analysis begins with `run_generic_realtime_analysis()` in the same [`commons.py`](https://github.com/call518/logsentinelai/blob/main/commons.py) module (lines 18-23). Rather than processing a static list, this function creates a `RealtimeLogMonitor` instance from [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py) and enters an infinite `while True` loop. The monitor polls for new data at intervals defined in `realtime_config["polling_interval"]`, maintaining a line buffer that captures only newly appended lines since the last check.

## Configuration and File Handling

The [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py) file establishes the default `ANALYSIS_MODE` as `"batch"` (lines 39-84), though the CLI parser in [`src/logsentinelai/cli.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/cli.py) (lines 61-84) can override this via the `--mode` flag.

**Batch mode** uses `analysis_mode="batch"` with a configuration containing `access_mode` (local or SSH) and `chunk_size`. It opens each target file once, reads sequentially from beginning to end, and closes the handle when complete.

**Realtime mode** uses `analysis_mode="realtime"` with additional configuration keys for polling intervals, sampling thresholds, and pending-chunk timeouts under the `realtime_config` section. The `RealtimeLogMonitor` class (lines 14-49 and 124-146 in [`monitoring.py`](https://github.com/call518/logsentinelai/blob/main/monitoring.py)) tracks file inodes and sizes to detect rotation or truncation events, resetting its buffer when logs roll over.

## Processing Loop Comparison

The execution flow differs fundamentally between the two modes.

**`run_generic_batch_analysis()`** implements a simple `for` loop over the expanded `log_files` list. For each file, it calls `_process_file_streaming_batch()` to handle chunks sequentially, releasing resources immediately after the final byte is processed.

**`run_generic_realtime_analysis()`** implements an infinite polling loop that calls `monitor.get_new_log_chunks()` to retrieve fresh data, builds an LLM prompt for the new content, executes the model call, and marks chunks as processed via `monitor.mark_chunk_processed()`. The loop sleeps for the configured polling duration between iterations, continuing until the process receives an interrupt signal.

## Practical Usage Examples

### Command-Line Batch Analysis

To analyze historic Apache access logs in one-off mode:

```bash
logsentinelai-linux-system \
    --mode batch \
    --log-path "/var/log/apache*access.log" \
    --chunk-size 500

```

This triggers `run_generic_batch_analysis()` and processes all matching files before exiting.

### Command-Line Realtime Monitoring

To continuously monitor a system log with sampling enabled:

```bash
logsentinelai-linux-system \
    --mode realtime \
    --log-path "/var/log/syslog" \
    --chunk-size 200 \
    --only-sampling-mode

```

This activates the `RealtimeLogMonitor` and enters the infinite analysis loop.

### Programmatic Integration

Both modes can be invoked directly from Python code:

```python
from logsentinelai.core.commons import run_generic_batch_analysis, run_generic_realtime_analysis
from logsentinelai.analyzers.httpd_access import HttpdAccessLogSchema, HTTPD_ACCESS_PROMPT

# Batch: Process all historic logs

run_generic_batch_analysis(
    log_type="httpd_access",
    analysis_schema_class=HttpdAccessLogSchema,
    prompt_template=HTTPD_ACCESS_PROMPT,
    analysis_title="Apache Access Log (Batch)",
    log_path="/var/log/apache*access.log",
    chunk_size=500,
)

# Realtime: Monitor live log stream

run_generic_realtime_analysis(
    log_type="httpd_access",
    analysis_schema_class=HttpdAccessLogSchema,
    prompt_template=HTTPD_ACCESS_PROMPT,
    analysis_title="Apache Access Log (Realtime)",
    log_path="/var/log/apache_access.log",
    chunk_size=200,
)

```

## Summary

- **Batch mode** is designed for finite, forensic analysis of static log files, utilizing `run_generic_batch_analysis()` in [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py) to process complete file lists in chunks before terminating with a summary report.
- **Realtime mode** provides indefinite, tail-style monitoring through `run_generic_realtime_analysis()` and the `RealtimeLogMonitor` class in [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py), which detects rotation, buffers new lines, and polls at configurable intervals.
- Both modes share common LLM initialization and prompt-building infrastructure, but differ in their configuration schemas, with realtime mode requiring additional parameters for polling and sampling thresholds in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py).
- **Batch processing** suits periodic security audits and back-fill operations, while **realtime processing** enables live alerting and automated incident response on active systems.

## Frequently Asked Questions

### Can I switch from batch to realtime mode without restarting LogSentinelAI?

No, the analysis mode is determined at startup when `get_analysis_config()` is invoked in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py). While the default `ANALYSIS_MODE` is `"batch"`, you must specify `--mode realtime` via CLI arguments processed in [`src/logsentinelai/cli.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/cli.py) lines 61-84 to activate the realtime monitoring loop. Changing modes requires restarting the process with the appropriate flags.

### How does realtime mode handle log rotation?

The `RealtimeLogMonitor` class in [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py) tracks file inodes and sizes to detect rotation or truncation events. When rotation is detected (lines 124-146), the monitor resets its internal buffer and file pointer, then continues reading from the new file instance. This ensures the infinite loop in `run_generic_realtime_analysis()` does not lose data during log rollover events.

### Is the chunk size parameter used differently between the two modes?

Yes. In **batch mode**, `chunk_size` controls memory consumption when reading large static files through `_process_file_streaming_batch()`, limiting how many lines are held in memory before LLM processing. In **realtime mode**, the same parameter defines the sampling threshold for the `RealtimeLogMonitor` buffer, determining how many pending lines must accumulate before triggering an LLM analysis call.

### Can realtime mode analyze multiple files simultaneously like batch mode?

No. While **batch mode** supports wildcard expansion and multiple files via `run_generic_batch_analysis()`, **realtime mode** is designed to monitor a single growing log file (or remote file via SSH) through one `RealtimeLogMonitor` instance. The realtime architecture focuses on tail-style monitoring of individual active logs for continuous security alerting rather than bulk multi-file processing.