How to Configure Log Chunk Size for Optimal Performance in LogSentinelAI

Set the CHUNK_SIZE_DEFAULT environment variable for global configuration, use log-type specific variables like CHUNK_SIZE_HTTPD_ACCESS for granular control, or pass --chunk-size to individual analyzer CLI commands to tune how many lines are batched per LLM request.

LogSentinelAI processes logs in batched segments to optimize LLM analysis throughput and cost. Configuring the log chunk size determines how many consecutive lines are grouped into a single inference request, directly impacting latency, token usage, and detection accuracy. The call518/logsentinelai repository provides multiple configuration layers to adjust this behavior without modifying source code.

Why Chunk Size Affects LogSentinelAI Performance

LogSentinelAI reads logs in chunks—groups of consecutive lines sent together to the LLM for analysis. The chunk size creates a direct trade-off between responsiveness and context depth.

Smaller chunks reduce per-request latency and memory buffers but increase total API calls and token overhead from repeated prompts. Larger chunks improve cross-line pattern detection and reduce request count, yet each inference takes longer and consumes more memory while building the batch. The default value of 10 lines per chunk suits typical web server logs, but optimal settings vary by log volume and complexity.

Where Chunk Size Is Defined in the Source Code

The chunk size configuration propagates through three core modules before reaching the analysis pipeline.

Configuration Defaults

In src/logsentinelai/core/config.py, the system loads per-log-type defaults from environment variables and stores them in the LOG_CHUNK_SIZES dictionary. Lines 107-111 define the fallback logic that populates these defaults when specific environment variables are absent.

Runtime Implementation

The src/logsentinelai/core/monitoring.py module resolves the final chunk_size value and prints it on startup (lines 138-141). Subsequently, src/logsentinelai/core/commons.py propagates this value through the analysis pipeline and records the finalized configuration in debug logs (lines 432-440).

CLI Overrides

Individual analyzer modules such as src/logsentinelai/analyzers/httpd_access.py expose a --chunk-size argument (line 90) that overrides both environment variables and defaults for single executions.

How to Configure Log Chunk Size in LogSentinelAI

You can adjust the chunk size through three methods, listed here in order of precedence (CLI overrides take highest priority).

Set Global Defaults via Environment Variables

Define CHUNK_SIZE_DEFAULT before launching LogSentinelAI to apply a single chunk size across all log types:

export CHUNK_SIZE_DEFAULT=20   # 20 lines per chunk

If this variable is unset, the system falls back to hardcoded per-type defaults in config.py.

Configure Per-Log-Type Chunk Sizes

For granular control, set environment variables specific to each analyzer type. These populate the LOG_CHUNK_SIZES mapping in src/logsentinelai/core/config.py:

• CHUNK_SIZE_HTTPD_ACCESS – Apache access logs
• CHUNK_SIZE_HTTPD_SERVER – Apache server logs
• CHUNK_SIZE_LINUX_SYSTEM – Linux system logs
• CHUNK_SIZE_GENERAL_LOG – Generic logs

Example configuration:

export CHUNK_SIZE_HTTPD_ACCESS=15
export CHUNK_SIZE_LINUX_SYSTEM=8

Override via CLI Arguments

When running a specific analyzer, pass --chunk-size to override all other settings for that execution:

python -m logsentinelai.analyzers.httpd_access \
    --log-path sample-logs/access-10k.log \
    --chunk-size 25

This argument takes precedence over environment variables and global defaults.

Verify Your Configuration

Upon startup, LogSentinelAI prints the active chunk size:

Chunk Size: 10 lines

With LOG_LEVEL=DEBUG enabled, the system logs the finalized state:

Final monitor state - access_mode: realtime, log_path: sample-logs/access-10k.log, chunk_size: 10

Performance Tuning Recommendations

Select chunk sizes based on your operational constraints and log characteristics:

• Low-volume logs (< 5k lines total): Use 5–10 lines. This minimizes latency for small datasets where request overhead is negligible.
• High-frequency real-time streaming: Use 8–12 lines. This balances timely alerting with manageable memory usage during continuous ingestion.
• Large heterogeneous logs requiring context: Use 20–30 lines. Larger batches provide the LLM sufficient surrounding lines to detect multi-line events and stack traces.
• Cost-sensitive environments: Use 5–8 lines. While this increases request count, it reduces total tokens consumed per analysis session.

Experiment by adjusting the environment variable, running a short test with python -m logsentinelai.analyzers.httpd_access, and observing the processing time reported in the console.

Summary

• Chunk size controls how many log lines are batched into each LLM request in LogSentinelAI.
• Configure global defaults with CHUNK_SIZE_DEFAULT or per-log-type variables like CHUNK_SIZE_HTTPD_ACCESS in src/logsentinelai/core/config.py.
• Override any setting temporarily using the --chunk-size CLI argument available in analyzer modules such as src/logsentinelai/analyzers/httpd_access.py.
• Verify active configuration via startup output in src/logsentinelai/core/monitoring.py or debug logs in src/logsentinelai/core/commons.py.
• Tune between 5–30 lines based on latency requirements, log volume, and LLM cost constraints.

Frequently Asked Questions

What is the default log chunk size in LogSentinelAI?

The default chunk size is 10 lines per chunk, which is defined in the configuration loader at src/logsentinelai/core/config.py. This value applies to most log types unless overridden by environment variables or CLI arguments.

How do I change the chunk size for a single analyzer run without affecting global settings?

Use the --chunk-size argument when invoking the specific analyzer module. For example, python -m logsentinelai.analyzers.httpd_access --chunk-size 15 will use 15 lines per chunk for that execution only, ignoring any environment variables.

Will increasing the chunk size reduce my LLM API costs?

Not necessarily. While larger chunks reduce the number of API requests, they may increase total token consumption because each request contains more lines. For cost-sensitive deployments, smaller chunks (5–8 lines) often prove more economical despite higher request counts, as they minimize repeated prompt tokens.

Where does LogSentinelAI log the currently active chunk size?

The system prints the active chunk size to the console on startup via src/logsentinelai/core/monitoring.py (lines 138-141). Additionally, when debug logging is enabled, src/logsentinelai/core/commons.py (lines 432-440) writes the finalized configuration including the chunk size to the application logs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}