Understanding the Difference Between Batch and Realtime Analysis Modes in LogSentinelAI

Batch mode processes static log files in finite, chunked streams via run_generic_batch_analysis(), while realtime mode runs an infinite monitoring loop through run_generic_realtime_analysis() and RealtimeLogMonitor to analyze new lines as they are written.

The LogSentinelAI open-source repository provides dual execution paths for log analysis, allowing security teams to choose between batch and realtime analysis modes depending on operational requirements. Understanding the difference between batch and realtime analysis modes in LogSentinelAI is essential for selecting the correct pipeline for forensic investigations versus continuous security monitoring.

Core Architectural Differences

Batch Mode Implementation

In src/logsentinelai/core/commons.py, the run_generic_batch_analysis() function (lines 53-58) serves as the entry point for batch processing. This mode expands wildcard patterns like /var/log/apache*.log into discrete file lists, then iterates through each file using _process_file_streaming_batch(). The function streams content in configurable chunks until all files are exhausted, then prints a summary and terminates.

Realtime Mode Implementation

Realtime analysis begins with run_generic_realtime_analysis() in the same commons.py module (lines 18-23). Rather than processing a static list, this function creates a RealtimeLogMonitor instance from src/logsentinelai/core/monitoring.py and enters an infinite while True loop. The monitor polls for new data at intervals defined in realtime_config["polling_interval"], maintaining a line buffer that captures only newly appended lines since the last check.

Configuration and File Handling

The src/logsentinelai/core/config.py file establishes the default ANALYSIS_MODE as "batch" (lines 39-84), though the CLI parser in src/logsentinelai/cli.py (lines 61-84) can override this via the --mode flag.

Batch mode uses analysis_mode="batch" with a configuration containing access_mode (local or SSH) and chunk_size. It opens each target file once, reads sequentially from beginning to end, and closes the handle when complete.

Realtime mode uses analysis_mode="realtime" with additional configuration keys for polling intervals, sampling thresholds, and pending-chunk timeouts under the realtime_config section. The RealtimeLogMonitor class (lines 14-49 and 124-146 in monitoring.py) tracks file inodes and sizes to detect rotation or truncation events, resetting its buffer when logs roll over.

Processing Loop Comparison

The execution flow differs fundamentally between the two modes.

run_generic_batch_analysis() implements a simple for loop over the expanded log_files list. For each file, it calls _process_file_streaming_batch() to handle chunks sequentially, releasing resources immediately after the final byte is processed.

run_generic_realtime_analysis() implements an infinite polling loop that calls monitor.get_new_log_chunks() to retrieve fresh data, builds an LLM prompt for the new content, executes the model call, and marks chunks as processed via monitor.mark_chunk_processed(). The loop sleeps for the configured polling duration between iterations, continuing until the process receives an interrupt signal.

Practical Usage Examples

Command-Line Batch Analysis

To analyze historic Apache access logs in one-off mode:

logsentinelai-linux-system \
    --mode batch \
    --log-path "/var/log/apache*access.log" \
    --chunk-size 500

This triggers run_generic_batch_analysis() and processes all matching files before exiting.

Command-Line Realtime Monitoring

To continuously monitor a system log with sampling enabled:

logsentinelai-linux-system \
    --mode realtime \
    --log-path "/var/log/syslog" \
    --chunk-size 200 \
    --only-sampling-mode

This activates the RealtimeLogMonitor and enters the infinite analysis loop.

Programmatic Integration

Both modes can be invoked directly from Python code:

from logsentinelai.core.commons import run_generic_batch_analysis, run_generic_realtime_analysis
from logsentinelai.analyzers.httpd_access import HttpdAccessLogSchema, HTTPD_ACCESS_PROMPT

# Batch: Process all historic logs

run_generic_batch_analysis(
    log_type="httpd_access",
    analysis_schema_class=HttpdAccessLogSchema,
    prompt_template=HTTPD_ACCESS_PROMPT,
    analysis_title="Apache Access Log (Batch)",
    log_path="/var/log/apache*access.log",
    chunk_size=500,
)

# Realtime: Monitor live log stream

run_generic_realtime_analysis(
    log_type="httpd_access",
    analysis_schema_class=HttpdAccessLogSchema,
    prompt_template=HTTPD_ACCESS_PROMPT,
    analysis_title="Apache Access Log (Realtime)",
    log_path="/var/log/apache_access.log",
    chunk_size=200,
)

Summary

• Batch mode is designed for finite, forensic analysis of static log files, utilizing run_generic_batch_analysis() in src/logsentinelai/core/commons.py to process complete file lists in chunks before terminating with a summary report.
• Realtime mode provides indefinite, tail-style monitoring through run_generic_realtime_analysis() and the RealtimeLogMonitor class in src/logsentinelai/core/monitoring.py, which detects rotation, buffers new lines, and polls at configurable intervals.
• Both modes share common LLM initialization and prompt-building infrastructure, but differ in their configuration schemas, with realtime mode requiring additional parameters for polling and sampling thresholds in src/logsentinelai/core/config.py.
• Batch processing suits periodic security audits and back-fill operations, while realtime processing enables live alerting and automated incident response on active systems.

Frequently Asked Questions

Can I switch from batch to realtime mode without restarting LogSentinelAI?

No, the analysis mode is determined at startup when get_analysis_config() is invoked in src/logsentinelai/core/config.py. While the default ANALYSIS_MODE is "batch", you must specify --mode realtime via CLI arguments processed in src/logsentinelai/cli.py lines 61-84 to activate the realtime monitoring loop. Changing modes requires restarting the process with the appropriate flags.

How does realtime mode handle log rotation?

The RealtimeLogMonitor class in src/logsentinelai/core/monitoring.py tracks file inodes and sizes to detect rotation or truncation events. When rotation is detected (lines 124-146), the monitor resets its internal buffer and file pointer, then continues reading from the new file instance. This ensures the infinite loop in run_generic_realtime_analysis() does not lose data during log rollover events.

Is the chunk size parameter used differently between the two modes?

Yes. In batch mode, chunk_size controls memory consumption when reading large static files through _process_file_streaming_batch(), limiting how many lines are held in memory before LLM processing. In realtime mode, the same parameter defines the sampling threshold for the RealtimeLogMonitor buffer, determining how many pending lines must accumulate before triggering an LLM analysis call.

Can realtime mode analyze multiple files simultaneously like batch mode?

No. While batch mode supports wildcard expansion and multiple files via run_generic_batch_analysis(), realtime mode is designed to monitor a single growing log file (or remote file via SSH) through one RealtimeLogMonitor instance. The realtime architecture focuses on tail-style monitoring of individual active logs for continuous security alerting rather than bulk multi-file processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}