# How to Optimize Agent Performance and Token Usage in Codebuff: 9 Production Techniques

> Discover 9 production techniques to optimize agent performance and token usage in Codebuff. Learn budget-aware tooling, file-tree pruning, and cached token counting for fast, relevant results within LLM limits.

- Repository: [Codebuff/codebuff](https://github.com/CodebuffAI/codebuff)
- Tags: best-practices
- Published: 2026-03-09

---

**Codebuff optimizes agent performance and token usage through budget-aware tooling, intelligent file-tree pruning, and cached token counting to stay within LLM provider limits while maintaining fast, relevant results.**

Codebuff is an agent-driven AI coding framework that must operate within strict token limits imposed by LLM providers. The repository implements a layered optimization strategy that balances runtime latency, cost, and output quality through precise token budgeting and intelligent content truncation.

## Token-Budget Aware Tool Parameters

Every tool capable of returning large content blobs accepts explicit token ceilings. In [`agents/types/tools.ts`](https://github.com/CodebuffAI/codebuff/blob/main/agents/types/tools.ts), the `ReadSubtreeParams` interface defines a `maxTokens` field that hard-caps returned text.

The implementation in [`packages/agent-runtime/src/tools/handlers/tool/read-subtree.ts`](https://github.com/CodebuffAI/codebuff/blob/main/packages/agent-runtime/src/tools/handlers/tool/read-subtree.ts) passes this budget directly into the truncation engine:

```typescript
const { paths, maxTokens } = toolCall.input
const tokenBudget = maxTokens
const { printedTree, tokenCount, truncationLevel } =
  truncateFileTreeBasedOnTokenBudget({
    fileContext: subctx,
    tokenBudget,
    logger,
  })

```

## Effort-Based Performance Tuning

Instead of manual token counting, developers supply an `effort` level—`high`, `medium`, `low`, `minimal`, or `none`—as defined in [`agents/types/agent-definition.ts`](https://github.com/CodebuffAI/codebuff/blob/main/agents/types/agent-definition.ts). The runtime translates these coarse settings into internal token budgets, allowing callers to select performance/quality trade-offs without calculating limits manually.

When `effort: "low"` is specified, the system reduces the number of files and variables emitted, cutting both latency and API costs while still returning actionable context.

## Three-Stage File-Tree Pruning Algorithm

The core optimization lives in [`packages/agent-runtime/src/system-prompt/truncate-file-tree.ts`](https://github.com/CodebuffAI/codebuff/blob/main/packages/agent-runtime/src/system-prompt/truncate-file-tree.ts). The `truncateFileTreeBasedOnTokenBudget` function implements a progressive truncation strategy:

1. **Remove unimportant files** – Build artifacts, minified assets, and cache directories are eliminated first.
2. **Drop low-value tokens** – Variables and imports are pruned based on a scoring map that identifies high-cost, low-relevance tokens.
3. **Depth-based removal** – If the tree still exceeds budget, the algorithm prunes deepest directories first until the limit is satisfied.

This ensures the most relevant source code remains while discarding expendable metadata.

## Fast Token Counting with LRU Caching

Repeated tokenization during pruning loops would cripple performance. The `countTokens` helper in [`packages/agent-runtime/src/util/token-counter.ts`](https://github.com/CodebuffAI/codebuff/blob/main/packages/agent-runtime/src/util/token-counter.ts) solves this using the `gpt-tokenizer` library with an Anthropic-specific fudge factor for accuracy.

The system maintains a global `LRUCache` with a capacity of 1,000 entries. Strings longer than 100 characters are memoized, preventing redundant encoding passes during iterative tree trimming.

## Statistical Sampling for Rapid Estimation

When token budgets are tight, the algorithm optimizes further by sampling 30 random files to estimate average token cost per file. This statistical approach, found in the sampling logic of `truncateFileTreeBasedOnTokenBudget`, dramatically reduces the number of full-tree token-count passes required to make truncation decisions.

## Cost-Aware System Prompt Assembly

The system prompt construction in [`packages/agent-runtime/src/system-prompt/search-system-prompt.ts`](https://github.com/CodebuffAI/codebuff/blob/main/packages/agent-runtime/src/system-prompt/search-system-prompt.ts) implements top-down budgeting. The runtime calculates a global ceiling—500,000 tokens for production workloads or 64,000 for "lite" mode—then allocates specific quotas for messages, file-tree content, and miscellaneous sections.

This guarantees the final assembled prompt never exceeds provider limits, regardless of repository size.

## API Compatibility Layer

In [`web/src/llm-api/openai.ts`](https://github.com/CodebuffAI/codebuff/blob/main/web/src/llm-api/openai.ts), the framework normalizes OpenAI-style `max_tokens` parameters to `max_completion_tokens` automatically. This conversion preserves user-specified token ceilings while maintaining compatibility with varying provider APIs, ensuring consistent limit enforcement across different LLM backends.

## Graceful Degradation on Budget Exhaustion

If the truncation algorithm cannot satisfy the token budget after all pruning stages, the system logs a warning via the logger and returns the best-effort tree produced during the final iteration. This prevents hard crashes when encountering edge-case repository structures or extremely tight constraints.

## Practical Implementation Examples

### Capping Tool Output with maxTokens

```typescript
// Tool call with explicit token budget
{
  "tool_name": "read_subtree",
  "input": {
    "paths": ["src"],
    "maxTokens": 50000
  }
}

```

### Using Effort Levels for Automatic Sizing

```typescript
{
  "tool_name": "read_subtree",
  "input": {
    "paths": ["src"],
    "effort": "low"
  }
}

```

### Manual Truncation for Custom Workflows

```typescript
import { truncateFileTreeBasedOnTokenBudget } from '@codebuff/agent-runtime/system-prompt'

const result = truncateFileTreeBasedOnTokenBudget({
  fileContext,
  tokenBudget: 30000,
  logger,
})

console.log(result.truncationLevel) // 'unimportant-files' | 'tokens' | 'depth-based'
console.log(result.tokenCount)      // Actual tokens in the pruned tree

```

### Leveraging Cached Token Counting

```typescript
import { countTokens } from '@codebuff/agent-runtime/util/token-counter'

const tokens = countTokens(`function optimize() { return true; }`)
// Subsequent identical calls hit the LRU cache instantly

```

## Summary

- **Token-budget aware tools** accept explicit `maxTokens` parameters in [`agents/types/tools.ts`](https://github.com/CodebuffAI/codebuff/blob/main/agents/types/tools.ts) to cap output size at the source.
- **Effort-based sizing** provides coarse-grained performance control without manual token arithmetic.
- **Three-stage pruning** removes unimportant files, low-value tokens, and deep directories progressively in [`truncate-file-tree.ts`](https://github.com/CodebuffAI/codebuff/blob/main/truncate-file-tree.ts).
- **Cached token counting** via `countTokens` in [`token-counter.ts`](https://github.com/CodebuffAI/codebuff/blob/main/token-counter.ts) eliminates redundant encoding with a 1,000-entry LRU cache.
- **Statistical sampling** of 30 random files estimates costs rapidly, minimizing full-tree passes.
- **System prompt budgeting** pre-allocates token quotas for messages and file-trees in [`search-system-prompt.ts`](https://github.com/CodebuffAI/codebuff/blob/main/search-system-prompt.ts).
- **API normalization** converts `max_tokens` to `max_completion_tokens` in [`openai.ts`](https://github.com/CodebuffAI/codebuff/blob/main/openai.ts) for cross-provider compatibility.
- **Graceful degradation** returns best-effort results when strict budgets cannot be met, preventing runtime failures.

## Frequently Asked Questions

### How does Codebuff prevent agents from exceeding LLM token limits?

Codebuff implements a hierarchical budgeting system. The `truncateFileTreeBasedOnTokenBudget` function in [`packages/agent-runtime/src/system-prompt/truncate-file-tree.ts`](https://github.com/CodebuffAI/codebuff/blob/main/packages/agent-runtime/src/system-prompt/truncate-file-tree.ts) progressively prunes repository content through three stages—removing unimportant files, scoring and dropping low-value tokens, and finally truncating deep directories—until the content fits within the specified `maxTokens` or derived effort-based budget.

### What is the difference between using maxTokens and effort parameters?

The `maxTokens` parameter in [`agents/types/tools.ts`](https://github.com/CodebuffAI/codebuff/blob/main/agents/types/tools.ts) specifies an exact token ceiling, while `effort` levels (`high`, `medium`, `low`, `minimal`, `none`) defined in [`agents/types/agent-definition.ts`](https://github.com/CodebuffAI/codebuff/blob/main/agents/types/agent-definition.ts) translate to internal token budgets automatically. Use `maxTokens` for precise control when you know the exact limit; use `effort` for coarse performance tuning without manual calculations.

### How does Codebuff optimize token counting performance during large file operations?

The framework uses a cached `countTokens` utility in [`packages/agent-runtime/src/util/token-counter.ts`](https://github.com/CodebuffAI/codebuff/blob/main/packages/agent-runtime/src/util/token-counter.ts) that memoizes results for strings over 100 characters in a global `LRUCache` (capacity 1,000). Additionally, when trimming file trees, the algorithm samples 30 random files to estimate average token costs statistically, avoiding expensive full-tree tokenization during every iteration.

### What happens if a repository is too large to fit within the specified token budget?

If the three-stage pruning algorithm cannot reduce the file tree below the token threshold, the system logs a warning and returns the best-effort tree produced at the final truncation stage. This graceful degradation, implemented at the end of [`truncate-file-tree.ts`](https://github.com/CodebuffAI/codebuff/blob/main/truncate-file-tree.ts), ensures the agent continues operating rather than crashing when encountering extreme repository sizes or unusually tight constraints.