Context Budget Resolution in Forge: How Token Allocation Works Across Ollama, Llama-Server, and Llamafile Backends

Context budget resolution determines the maximum tokens a model can process per request by combining backend-specific discovery mechanisms with configurable allocation strategies defined in the BudgetMode enum.

The forge framework abstracts LLM interaction across multiple local inference backends, but each handles context window sizing differently. Understanding how forge unifies these disparate approaches reveals how to optimize memory usage and inference speed for your specific hardware configuration.

The Two Dimensions of Context Budget

Context budget resolution operates along two axes defined in src/forge/server.py: the Backend dimension (ollama, llamaserver, llamafile) and the BudgetMode dimension (BACKEND, MANUAL, FORGE_FULL, FORGE_FAST).

The BudgetMode enum (lines 26-33) declares four distinct strategies:

• BACKEND – Respect the backend's native default or auto-tuned value
• MANUAL – Enforce a specific token limit provided by the caller
• FORGE_FULL – Utilize the maximum available context for the hardware
• FORGE_FAST – Trade half the available context for improved generation speed

The core resolution logic lives in ServerManager.resolve_budget (lines 52-90) and ServerManager.start_with_budget (lines 91-124), which orchestrate server startup and context negotiation.

Ollama Backend Resolution

For Ollama backends, forge relies on GPU VRAM detection rather than server process management. The _ollama_vram_tier_budget function (lines 84-94) maps detected hardware to default budgets: 4,096 tokens for low VRAM, 32,768 for mid-range, and 262,144 for high-end GPUs (≥48GB).

BudgetMode behavior for Ollama:

• BACKEND / FORGE_FULL – Returns the full VRAM-tier budget (32,768 on a 24GB GPU). No server process is started.
• FORGE_FAST – Returns half the full budget (full // 2), reducing KV cache memory pressure.
• MANUAL – Returns the exact manual_tokens value specified by the caller, bypassing hardware detection.

Llama-Server and Llamafile Backend Resolution

Unlike Ollama, these backends expose a /props endpoint reporting the actual context size configured via the -c flag. resolve_budget uses get_server_context() (lines 85-89) to read this value after server startup.

Manual Mode

When budget_mode=BudgetMode.MANUAL, the server starts with -c <manual_tokens>. The method then queries /props to confirm the server accepted the override, returning that verified value.

Forge Full Mode

FORGE_FULL starts the server without specifying -c, allowing the binary to auto-tune to the maximum context supported by the model and hardware. The resolved budget equals the value reported by /props.

Forge Fast Mode

FORGE_FAST implements a two-phase startup dance (lines 91-124):

1. Phase 1 – Start without -c and read the full context from /props
2. Phase 2 – Calculate half_total = full // 2 (adjusting for slots if applicable), restart with -c half_total, and verify via /props

This yields a budget trading maximum sequence length for reduced memory usage and faster token generation.

Backend Mode

For these servers, BACKEND behaves identically to FORGE_FULL, using the auto-tuned full context.

Handling Multi-Slot Configurations

When running multiple parallel slots (n_slots > 1), context calculation differs based on KV cache architecture:

Non-unified KV cache (kv_unified=False) – The /props endpoint reports context per-slot. start_with_budget multiplies this by n_slots when calculating the total budget for FORGE_FAST halving (lines 61-64).

Unified KV cache (kv_unified=True) – /props reports the total shared context across all slots; no multiplication occurs.

The integer returned by start_with_budget always reflects the budget passed to ContextManager, representing either per-slot (default) or total context depending on the kv_unified flag (docstring lines 15-20).

Integration with setup_backend

The high-level setup_backend function orchestrates the complete resolution flow:

1. Instantiates ServerManager with the selected backend
2. Calls start_with_budget to resolve the budget according to the rules above
3. For Ollama, wires the budget into the client via client.set_num_ctx(budget)
4. Constructs a ContextManager with budget_tokens=budget (docstring lines 58-81)

This provides a unified interface where downstream code receives a validated integer representing available tokens, regardless of backend complexity.

Practical Examples

Resolve a full budget on Ollama using auto-detected VRAM tiers:

from forge import BudgetMode, setup_backend
from forge.clients.ollama import OllamaClient

client = OllamaClient(model="llama3")
server, ctx = await setup_backend(
    backend="ollama",
    model="llama3",
    budget_mode=BudgetMode.FORGE_FULL,
    client=client,
)
print(f"Ollama full budget: {ctx.budget_tokens} tokens")

# Output: 32768 (on 24GB GPU) or 262144 (on 48GB+ GPU)

Force a specific context size on llama-server using manual mode:

from forge import BudgetMode, setup_backend

server, ctx = await setup_backend(
    backend="llamaserver",
    gguf_path="/models/llama3.gguf",
    budget_mode=BudgetMode.MANUAL,
    manual_tokens=8192,
)
print(f"Manual budget: {ctx.budget_tokens} tokens")

# Output: 8192 (verified via /props endpoint)

Optimize for speed with multiple slots on llamafile:

from forge import BudgetMode, setup_backend

server, ctx = await setup_backend(
    backend="llamafile",
    gguf_path="/models/phi3.gguf",
    n_slots=4,
    kv_unified=False,
    budget_mode=BudgetMode.FORGE_FAST,
)
print(f"Fast budget per slot: {ctx.budget_tokens} tokens")

# Calculation: (full_context * 4 slots) // 2

Use unified KV cache for shared context across slots:

from forge import BudgetMode, setup_backend

server, ctx = await setup_backend(
    backend="llamafile",
    gguf_path="/models/phi3.gguf",
    n_slots=4,
    kv_unified=True,
    budget_mode=BudgetMode.FORGE_FAST,
)
print(f"Fast total budget: {ctx.budget_tokens} tokens")

# Output: half of total shared context from /props

Summary

• Context budget resolution in forge combines backend-specific discovery (ollama via VRAM tiers, llamaserver/llamafile via /props) with four allocation strategies defined in BudgetMode.
• Ollama relies on _ollama_vram_tier_budget to map GPU memory to preset token limits (4096, 32768, or 262144), supporting full, fast (half), or manual modes.
• Llama-server and llamafile execute a startup-validation cycle where start_with_budget starts the process, queries /props via get_server_context(), and optionally restarts with adjusted -c flags for FORGE_FAST optimization.
• Multi-slot inference requires adjusting calculations based on kv_unified: non-unified configurations multiply per-slot context by n_slots before halving for fast mode, while unified configurations use the total directly.
• All resolution paths converge in setup_backend, which returns a ContextManager configured with a verified integer budget regardless of backend complexity.

Frequently Asked Questions

How does FORGE_FAST mode improve inference performance?

FORGE_FAST reduces the KV cache memory footprint by allocating half the available context window (calculated as full_context // 2). According to the implementation in src/forge/server.py (lines 91-124), this reduction decreases memory bandwidth pressure and cache management overhead, trading maximum sequence length for faster token generation speeds, particularly beneficial in multi-slot scenarios.

Can I override the automatic context detection on Ollama backends?

Yes, by specifying budget_mode=BudgetMode.MANUAL and providing the manual_tokens parameter to setup_backend. While Ollama normally relies on _ollama_vram_tier_budget to automatically select 4096, 32768, or 262144 tokens based on detected GPU VRAM, manual mode bypasses this hardware detection and sets num_ctx directly on the client via client.set_num_ctx().

Why does context calculation differ between unified and non-unified KV caches?

In non-unified KV cache configurations (kv_unified=False), each parallel slot maintains separate KV memory, so /props reports context per-slot and forge multiplies this by n_slots to calculate total capacity before halving for FORGE_FAST. In unified configurations (kv_unified=True), all slots share a single KV cache, meaning /props already reports the total shared context, requiring no multiplication adjustment during budget resolution (lines 61-64).

What happens if the server rejects my manual context size?

When using BudgetMode.MANUAL with llamaserver or llamafile backends, start_with_budget starts the server with -c <manual_tokens> and subsequently queries /props via get_server_context() to verify the actual allocated context. If the server binary caps the value below your request due to model or hardware limitations, the resolved budget reflects the server's reported maximum rather than the requested value, ensuring ContextManager never assumes more tokens than physically available.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}