# How to Configure Log Chunk Size for Optimal Performance in LogSentinelAI

> Optimize LogSentinelAI performance by configuring log chunk size. Learn to set CHUNK_SIZE_DEFAULT, log-type specific variables, or use CLI commands for efficient LLM requests.

- Repository: [JungJungIn/logsentinelai](https://github.com/call518/logsentinelai)
- Tags: performance
- Published: 2026-02-26

---

**Set the `CHUNK_SIZE_DEFAULT` environment variable for global configuration, use log-type specific variables like `CHUNK_SIZE_HTTPD_ACCESS` for granular control, or pass `--chunk-size` to individual analyzer CLI commands to tune how many lines are batched per LLM request.**

LogSentinelAI processes logs in batched segments to optimize LLM analysis throughput and cost. Configuring the **log chunk size** determines how many consecutive lines are grouped into a single inference request, directly impacting latency, token usage, and detection accuracy. The `call518/logsentinelai` repository provides multiple configuration layers to adjust this behavior without modifying source code.

## Why Chunk Size Affects LogSentinelAI Performance

LogSentinelAI reads logs in **chunks**—groups of consecutive lines sent together to the LLM for analysis. The chunk size creates a direct trade-off between responsiveness and context depth.

Smaller chunks reduce per-request latency and memory buffers but increase total API calls and token overhead from repeated prompts. Larger chunks improve cross-line pattern detection and reduce request count, yet each inference takes longer and consumes more memory while building the batch. The default value of **10 lines per chunk** suits typical web server logs, but optimal settings vary by log volume and complexity.

## Where Chunk Size Is Defined in the Source Code

The chunk size configuration propagates through three core modules before reaching the analysis pipeline.

### Configuration Defaults

In [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py), the system loads per-log-type defaults from environment variables and stores them in the `LOG_CHUNK_SIZES` dictionary. Lines 107-111 define the fallback logic that populates these defaults when specific environment variables are absent.

### Runtime Implementation

The [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py) module resolves the final `chunk_size` value and prints it on startup (lines 138-141). Subsequently, [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py) propagates this value through the analysis pipeline and records the finalized configuration in debug logs (lines 432-440).

### CLI Overrides

Individual analyzer modules such as [`src/logsentinelai/analyzers/httpd_access.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/analyzers/httpd_access.py) expose a `--chunk-size` argument (line 90) that overrides both environment variables and defaults for single executions.

## How to Configure Log Chunk Size in LogSentinelAI

You can adjust the chunk size through three methods, listed here in order of precedence (CLI overrides take highest priority).

### Set Global Defaults via Environment Variables

Define `CHUNK_SIZE_DEFAULT` before launching LogSentinelAI to apply a single chunk size across all log types:

```bash
export CHUNK_SIZE_DEFAULT=20   # 20 lines per chunk

```

If this variable is unset, the system falls back to hardcoded per-type defaults in [`config.py`](https://github.com/call518/logsentinelai/blob/main/config.py).

### Configure Per-Log-Type Chunk Sizes

For granular control, set environment variables specific to each analyzer type. These populate the `LOG_CHUNK_SIZES` mapping in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py):

- `CHUNK_SIZE_HTTPD_ACCESS` – Apache access logs
- `CHUNK_SIZE_HTTPD_SERVER` – Apache server logs  
- `CHUNK_SIZE_LINUX_SYSTEM` – Linux system logs
- `CHUNK_SIZE_GENERAL_LOG` – Generic logs

Example configuration:

```bash
export CHUNK_SIZE_HTTPD_ACCESS=15
export CHUNK_SIZE_LINUX_SYSTEM=8

```

### Override via CLI Arguments

When running a specific analyzer, pass `--chunk-size` to override all other settings for that execution:

```bash
python -m logsentinelai.analyzers.httpd_access \
    --log-path sample-logs/access-10k.log \
    --chunk-size 25

```

This argument takes precedence over environment variables and global defaults.

### Verify Your Configuration

Upon startup, LogSentinelAI prints the active chunk size:

```

Chunk Size: 10 lines

```

With `LOG_LEVEL=DEBUG` enabled, the system logs the finalized state:

```

Final monitor state - access_mode: realtime, log_path: sample-logs/access-10k.log, chunk_size: 10

```

## Performance Tuning Recommendations

Select chunk sizes based on your operational constraints and log characteristics:

- **Low-volume logs (< 5k lines total)**: Use **5–10 lines**. This minimizes latency for small datasets where request overhead is negligible.
- **High-frequency real-time streaming**: Use **8–12 lines**. This balances timely alerting with manageable memory usage during continuous ingestion.
- **Large heterogeneous logs requiring context**: Use **20–30 lines**. Larger batches provide the LLM sufficient surrounding lines to detect multi-line events and stack traces.
- **Cost-sensitive environments**: Use **5–8 lines**. While this increases request count, it reduces total tokens consumed per analysis session.

Experiment by adjusting the environment variable, running a short test with `python -m logsentinelai.analyzers.httpd_access`, and observing the processing time reported in the console.

## Summary

- **Chunk size** controls how many log lines are batched into each LLM request in LogSentinelAI.
- Configure global defaults with `CHUNK_SIZE_DEFAULT` or per-log-type variables like `CHUNK_SIZE_HTTPD_ACCESS` in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py).
- Override any setting temporarily using the `--chunk-size` CLI argument available in analyzer modules such as [`src/logsentinelai/analyzers/httpd_access.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/analyzers/httpd_access.py).
- Verify active configuration via startup output in [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py) or debug logs in [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py).
- Tune between 5–30 lines based on latency requirements, log volume, and LLM cost constraints.

## Frequently Asked Questions

### What is the default log chunk size in LogSentinelAI?

The default chunk size is **10 lines per chunk**, which is defined in the configuration loader at [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py). This value applies to most log types unless overridden by environment variables or CLI arguments.

### How do I change the chunk size for a single analyzer run without affecting global settings?

Use the `--chunk-size` argument when invoking the specific analyzer module. For example, `python -m logsentinelai.analyzers.httpd_access --chunk-size 15` will use 15 lines per chunk for that execution only, ignoring any environment variables.

### Will increasing the chunk size reduce my LLM API costs?

Not necessarily. While larger chunks reduce the number of API requests, they may increase total token consumption because each request contains more lines. For cost-sensitive deployments, smaller chunks (5–8 lines) often prove more economical despite higher request counts, as they minimize repeated prompt tokens.

### Where does LogSentinelAI log the currently active chunk size?

The system prints the active chunk size to the console on startup via [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py) (lines 138-141). Additionally, when debug logging is enabled, [`src/logsentinelai/core/commons.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/commons.py) (lines 432-440) writes the finalized configuration including the chunk size to the application logs.