How to Manage Token Usage and Optimize Costs for Cloud LLM Providers in LogSentinelAI

LogSentinelAI provides built-in token counting and configuration levers—LLM_MAX_TOKENS, LOG_CHUNK_SIZES, and sampling parameters—that let you monitor and cap cloud LLM costs without modifying core logic.

LogSentinelAI is an open-source log analysis framework that abstracts interactions with cloud LLM providers (OpenAI, Ollama, vLLM, Gemini) through a unified interface. Because token usage directly determines billing for these providers, the library ships with dedicated utilities to track and control token consumption. This guide explains how to leverage the token-counting architecture in src/logsentinelai/core/token_utils.py and the configuration system in src/logsentinelai/core/config.py to optimize your cloud LLM costs.

Understanding the Token Counting Architecture

LogSentinelAI estimates token consumption using the count_tokens helper in src/logsentinelai/core/token_utils.py. This function uses the tiktoken library to invoke tiktoken.encoding_for_model() when available, ensuring the estimate matches the provider’s billing units exactly. If tiktoken is not installed, it falls back to a deterministic word-count heuristic that provides a consistent, conservative upper bound.

The token counter is invoked automatically within the processing pipeline. In src/logsentinelai/core/commons.py, the process_log_chunk function calls count_tokens(review, llm_model) immediately after receiving the LLM response (lines 73–78). For real-time processing, it also counts prompt tokens using count_tokens(prompt, llm_model_name) (lines 52–56). These counts are logged as "Prompt tokens (approx.)" and "LLM response tokens (approx.)", giving you per-request visibility into consumption.

Configuration Levers for Cost Control

Cost optimization in LogSentinelAI relies on three configuration variables defined in src/logsentinelai/core/config.py. These are read by the LLM initializer in src/logsentinelai/core/llm.py (lines 20–36) and passed to the provider-specific client on every request.

Limiting Response Size with LLM_MAX_TOKENS

The LLM_MAX_TOKENS parameter (config.py line 37) sets the max_tokens argument passed to the cloud provider. This caps the size of the generated JSON response, preventing runaway generation that could spike costs. For budget-conscious deployments, set this to 512 or 256 for simple classification tasks.

Tuning Sampling Parameters

LLM_TEMPERATURE and LLM_TOP_P (config.py lines 35–36) control the randomness of the model output. Lower values (e.g., temperature=0.2, top_p=0.5) produce more deterministic, shorter responses, which often reduces token waste from verbose or repetitive generation.

Adjusting LOG_CHUNK_SIZES

The LOG_CHUNK_SIZES dictionary (config.py lines 43–48) defines how many log lines are bundled into a single prompt. Smaller chunks mean shorter prompts and fewer input tokens, though they may reduce context. A typical safe range is 5–20 lines per chunk, balancing cost and analysis quality.

Monitoring Token Usage in Processing Pipelines

Every analysis pipeline—batch, streaming batch, and real-time—automatically logs token counts via the process_log_chunk function in src/logsentinelai/core/commons.py. After the LLM returns a response, the code executes:

response_tokens = count_tokens(review, llm_model)
logger.info(f"LLM response tokens (approx.): {response_tokens}")

For real-time mode, the prompt is also counted before the call:

prompt_tokens = count_tokens(prompt, llm_model_name)
logger.info(f"Prompt tokens (approx.): {prompt_tokens}")

These log entries provide the raw data needed to calculate per-chunk costs using your provider’s price-per-1K-tokens rate.

Cost Optimization Workflow

Follow this workflow to minimize cloud LLM expenses without sacrificing analysis accuracy:

1. Set conservative limits in your environment file or /etc/logsentinelai.config. Cap LLM_MAX_TOKENS to 512 for cheap models like gpt-4o-mini, and set LLM_TEMPERATURE=0.2 to reduce verbosity.

2. Tune chunk sizes per log type. Start with LOG_CHUNK_SIZES={"linux_system": 8, "httpd_access": 10}. Smaller chunks reduce input tokens but retain enough context for classification.

3. Monitor token logs during a trial run. Parse the "Prompt tokens" and "LLM response tokens" entries to compute the per-chunk cost: (total_tokens / 1000) × price_per_1k.

4. Iterate dynamically. If logs show consistent truncation (responses hitting LLM_MAX_TOKENS), increase the limit or reduce chunk size. If token counts are consistently low, lower LLM_MAX_TOKENS to cut costs further.

Code Examples

Example 1: Configure Low-Cost Parameters

Set a strict token budget for OpenAI models via environment variables or a config file:

# .env or /etc/logsentinelai.config

LLM_PROVIDER=openai
LLM_MODEL_OPENAI=gpt-4o-mini
LLM_MAX_TOKENS=512
LLM_TEMPERATURE=0.2
LOG_CHUNK_SIZES={"linux_system": 8, "httpd_access": 10}

Example 2: Programmatically Estimate Costs

Use the internal count_tokens helper to calculate real-time expenses:

from logsentinelai.core.token_utils import count_tokens
from logsentinelai.core.config import LLM_MAX_TOKENS

def estimate_chunk_cost(prompt: str, response: str, price_per_1k: float) -> float:
    """Calculate USD cost for a single log chunk analysis."""
    prompt_tokens = count_tokens(prompt)
    resp_tokens = count_tokens(response)
    total_tokens = prompt_tokens + resp_tokens
    
    return (total_tokens / 1000.0) * price_per_1k

# Usage within a custom analyzer

cost = estimate_chunk_cost(prompt, review, price_per_1k=0.002)
print(f"Estimated chunk cost: ${cost:.5f}")

Example 3: Dynamic Chunk Size Adaptation

Adjust chunk sizes based on recent token consumption patterns:

from logsentinelai.core.config import LOG_CHUNK_SIZES

def adapt_chunk_size(log_type: str, recent_avg_tokens: int, target_tokens: int = 300) -> int:
    """Dynamically resize chunks to hit a target token count."""
    current_size = LOG_CHUNK_SIZES.get(log_type, 10)
    factor = max(1, int(target_tokens / max(1, recent_avg_tokens)))
    new_size = current_size * factor
    
    # Clamp to safe bounds

    return max(5, min(new_size, 30))

# Example: shrink chunks if we're using too many tokens

new_size = adapt_chunk_size("linux_system", recent_avg_tokens=450)
print(f"Adjusted chunk size: {new_size}")

Summary

• Token visibility: LogSentinelAI automatically counts prompt and response tokens using count_tokens in src/logsentinelai/core/token_utils.py, logging estimates for every chunk processed.
• Cost levers: Control spending via LLM_MAX_TOKENS, LLM_TEMPERATURE, LLM_TOP_P, and LOG_CHUNK_SIZES defined in src/logsentinelai/core/config.py.
• Provider abstraction: The initialize_llm_model function in src/logsentinelai/core/llm.py ensures all providers respect these limits.
• Optimization workflow: Set conservative defaults, monitor per-chunk token logs, calculate costs using provider pricing, and iteratively tune chunk sizes and max-token limits.

Frequently Asked Questions

How does LogSentinelAI count tokens without accessing the provider's API?

LogSentinelAI uses the tiktoken library to load the exact encoding for the target model (e.g., cl100k_base for GPT-4) via tiktoken.encoding_for_model(). If tiktoken is unavailable, it falls back to a deterministic word-count heuristic that provides a conservative estimate. This happens locally in src/logsentinelai/core/token_utils.py before or after the API call, so you get accurate billing estimates without extra network overhead.

Can I use different token limits for different log types?

While LLM_MAX_TOKENS is a global setting in src/logsentinelai/core/config.py, you can achieve per-log-type limits by adjusting LOG_CHUNK_SIZES. Smaller chunks naturally reduce both input tokens and the likelihood of hitting the max-tokens ceiling. For advanced use cases, you can instantiate separate analyzer instances with different environment configurations pointing to distinct config files, each with unique LLM_MAX_TOKENS values.

What happens if a response exceeds LLM_MAX_TOKENS?

When the LLM generates a response that hits the LLM_MAX_TOKENS limit, the provider truncates the output at that boundary. In LogSentinelAI, this typically results in an incomplete JSON structure that fails Pydantic validation in process_log_chunk within src/logsentinelai/core/commons.py. The error is logged, and the chunk is marked as failed. To avoid this, monitor your token logs; if you see frequent truncation errors, either increase LLM_MAX_TOKENS or decrease LOG_CHUNK_SIZES to give the model more room to generate complete responses within the limit.

Does LogSentinelAI support local LLMs to eliminate cloud token costs entirely?

Yes. The initialize_llm_model function in src/logsentinelai/core/llm.py supports provider values such as ollama and vllm. When configured to use a local endpoint, token counting still occurs for monitoring purposes, but no cloud billing is incurred. You can set LLM_PROVIDER=ollama and LLM_MODEL_OLLAMA=llama3.1 in your configuration to route all requests to a local instance while retaining the same token visibility and chunk-size optimization logic used for cloud providers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}