# How to Configure Sampling Mode for Realtime Log Monitoring in LogSentinelAI

> Learn how to configure sampling mode for realtime log monitoring in LogSentinelAI. Optimize performance by enabling sampling or setting an automatic threshold for incoming logs.

- Repository: [JungJungIn/logsentinelai](https://github.com/call518/logsentinelai)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Set `REALTIME_ONLY_SAMPLING_MODE=true` in your environment file to force sampling mode, or adjust `REALTIME_SAMPLING_THRESHOLD` to automatically switch to sampling when pending log lines exceed the configured limit.**

LogSentinelAI's real-time monitor processes log files in continuous chunks and supports two distinct modes: full retention and sampling mode. This guide covers how to configure sampling mode for realtime log monitoring using environment variables defined in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py) and processed by the monitor in [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py).

## Understanding Processing Modes

The real-time monitor operates in one of two modes depending on your configuration:

- **Full Mode**: Retains all incoming log lines until the configured chunk size is reached. This consumes more memory but preserves complete context.
- **Sampling Mode**: Retains only the most recent *N* lines (up to `chunk_size`), discarding older lines to conserve memory during high-volume periods.

According to the source code in [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py), the monitor evaluates the following logic to determine which mode to use:

```python
self.only_sampling_mode = self.realtime_config["only_sampling_mode"]
self.sampling_threshold = self.realtime_config["sampling_threshold"]

if self.only_sampling_mode:
    should_sample = True                     # always sampling

elif len(self.pending_lines) > self.sampling_threshold:
    should_sample = True                     # auto-switch when threshold exceeded

else:
    should_sample = False                    # full processing

```

## Key Environment Variables

Configuration is loaded at startup from `.env` or `/etc/logsentinelai.config` via the `REALTIME_CONFIG` dictionary in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py) (lines 90-98). The relevant variables for sampling mode are:

- **`REALTIME_ONLY_SAMPLING_MODE`** (default: `false`): When set to `true`, forces the monitor to permanently operate in sampling mode.
- **`REALTIME_SAMPLING_THRESHOLD`** (default: `100`): In full mode, automatically switches to sampling when pending lines exceed this count.
- **`REALTIME_CHUNK_PENDING_TIMEOUT`** (default: `1800`): Forces processing after this many seconds (default 30 minutes) regardless of chunk size.
- **`REALTIME_MAX_LINES_PER_BATCH`** (default: `50`): Upper bound of lines processed per poll cycle.
- **`REALTIME_POLLING_INTERVAL`** (default: `5`): Seconds between file system checks.
- **`REALTIME_BUFFER_TIME`** (default: `2`): Buffer time in seconds for line aggregation.

Note that changes to these variables require a monitor restart to take effect, as the configuration is read once at startup via the `apply_config()` function.

## Configuring Sampling Mode

### Option 1: Force Permanent Sampling Mode

To always run in sampling mode regardless of log volume:

1. Edit your `.env` file:

```dotenv
REALTIME_ONLY_SAMPLING_MODE=true
REALTIME_SAMPLING_THRESHOLD=100  # Ignored when only_sampling_mode is true

REALTIME_CHUNK_PENDING_TIMEOUT=300
REALTIME_MAX_LINES_PER_BATCH=50

```

2. Restart the monitor. The initialization banner in `monitoring.py::_print_initialization_info()` will display `MODE: SAMPLING-ONLY`.

### Option 2: Conditional Sampling with Threshold

To use full mode normally but switch to sampling during high-volume spikes:

```dotenv
REALTIME_ONLY_SAMPLING_MODE=false
REALTIME_SAMPLING_THRESHOLD=200   # Switch to sampling when >200 lines pending

REALTIME_CHUNK_PENDING_TIMEOUT=1800

```

### Option 3: Temporary CLI Override

Run a one-off analysis with sampling enabled without editing configuration files:

```bash
REALTIME_ONLY_SAMPLING_MODE=true REALTIME_SAMPLING_THRESHOLD=300 \
logsentinelai linux-system --mode realtime

```

## Programmatic Configuration

When using the Python API directly, you can override environment defaults by manipulating the configuration dictionary before instantiating the monitor:

```python
from logsentinelai.core.monitoring import create_realtime_monitor
from logsentinelai.core.config import get_analysis_config

# Build base configuration

custom_cfg = get_analysis_config(
    log_type="linux_system",
    analysis_mode="realtime",
    remote_mode="local",
    chunk_size=20,
)

# Force sampling mode programmatically

custom_cfg["realtime_config"]["only_sampling_mode"] = True
custom_cfg["realtime_config"]["sampling_threshold"] = 9999

# Create monitor with modified config

monitor = create_realtime_monitor(
    log_type="linux_system",
    remote_mode="local",
    ssh_config=None,
    remote_log_path=None,
)

# Process chunks

for chunk in monitor.get_new_log_chunks():
    print("Processing chunk:", chunk)

```

## Verifying Your Configuration

Upon startup, the monitor prints an initialization banner via `_print_initialization_info()` in [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py). Check the output for the active mode:

```

MODE: SAMPLING-ONLY    # If only_sampling_mode=true

MODE: FULL             # If processing all lines

```

If you do not see the expected mode, verify that your environment file is in the project root (for `.env`) or at `/etc/logsentinelai.config`, and ensure the monitor process was restarted after configuration changes.

## Summary

- **Sampling mode** in LogSentinelAI retains only the most recent lines up to `chunk_size`, while **full mode** retains all lines until the chunk is complete.
- Set `REALTIME_ONLY_SAMPLING_MODE=true` to force sampling permanently, or tune `REALTIME_SAMPLING_THRESHOLD` to trigger sampling automatically when pending lines exceed the limit.
- Configuration is loaded from environment variables via [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py) and evaluated in [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py) at startup.
- Changes to sampling configuration require a monitor restart to take effect.
- Verify the active mode by checking the initialization banner printed by `monitoring.py::_print_initialization_info()`.

## Frequently Asked Questions

### What is the difference between full mode and sampling mode in LogSentinelAI?

In **full mode**, the real-time monitor accumulates all incoming log lines until the chunk size is reached, preserving complete context but consuming more memory. In **sampling mode**, only the most recent *N* lines (where *N* equals the configured chunk size) are retained, and older lines are discarded to reduce memory footprint during high-volume logging.

### How do I know if sampling mode is currently active?

The monitor prints an initialization banner at startup via the `_print_initialization_info()` function in [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py). Look for `MODE: SAMPLING-ONLY` to confirm forced sampling mode, or `MODE: FULL` to confirm standard processing. If you see `SAMPLING-ONLY` while running with `only_sampling_mode=false`, it indicates the automatic threshold was triggered.

### Can I change sampling settings without restarting the monitor?

No. The `REALTIME_CONFIG` dictionary is built once at startup in [`src/logsentinelai/core/config.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/config.py) via the `apply_config()` function. Any changes to `REALTIME_ONLY_SAMPLING_MODE`, `REALTIME_SAMPLING_THRESHOLD`, or related variables require stopping and restarting the monitor process to reload the configuration from the environment.

### What happens when `REALTIME_SAMPLING_THRESHOLD` is exceeded in full mode?

When `only_sampling_mode` is `false` and the number of pending lines exceeds `REALTIME_SAMPLING_THRESHOLD` (default 100), the monitor automatically switches to sampling mode for that chunk. As implemented in [`src/logsentinelai/core/monitoring.py`](https://github.com/call518/logsentinelai/blob/main/src/logsentinelai/core/monitoring.py), the logic evaluates `len(self.pending_lines) > self.sampling_threshold` and sets `should_sample = True`, causing the monitor to retain only the most recent lines up to the chunk size instead of the full backlog.