# How the Forge Proxy Server Applies Guardrails Transparently

> Discover how the Forge proxy server transparently applies guardrails by executing safety mechanisms within its OpenAI-compatible front-end, delivering clean standard responses.

- Repository: [Antoine/forge](https://github.com/antoinezambelli/forge)
- Tags: internals
- Published: 2026-05-22

---

**The Forge proxy server applies guardrails transparently by executing all safety mechanisms—context compaction, rescue parsing, and synthetic tool injection—inside its OpenAI-compatible HTTP front-end, exposing only clean standard responses to clients.**

The `antoinezambelli/forge` repository implements a thin HTTP proxy that presents an OpenAI-compatible `POST /v1/chat/completions` endpoint while internally leveraging Forge's robust inference pipeline. When you send requests to this proxy server, it automatically applies guardrails transparently through its front-half processing layer, ensuring token budgets are respected and malformed outputs are corrected without exposing internal retry loops to the end user.

## Core Architecture and Request Flow

The proxy intercepts incoming OpenAI-style requests and processes them through Forge's *front-half* inference pipeline, reusing the same components as the `WorkflowRunner` but stopping before tool execution. This design is documented in **ADR-012 – OpenAI-Compatible Proxy Server**, which outlines the architectural split between front-half preprocessing and back-half execution.

### Request Handling and Message Conversion

Incoming requests hit `forge.proxy.handler.handle_chat_completions`, which performs the initial transformation. Located in [[`src/forge/proxy/handler.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/handler.py)](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/handler.py#L75-L115), this function:

- Converts OpenAI message objects to Forge-internal `Message` instances using `openai_to_messages`
- Extracts supplied tool schemas from the request body
- **Injects the synthetic `respond` tool** when tools are present but no explicit `respond` specification exists, ensuring the model remains in tool-calling mode

### Front-Half Inference Pipeline

After conversion, the handler delegates to `forge.core.inference.run_inference` ([[`src/forge/core/inference.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/core/inference.py)](https://github.com/antoinezambelli/forge/blob/main/src/forge/core/inference.py#L107-L124)). This shared function performs several preprocessing steps before contacting the backend:

- **Context compaction** via `ContextManager.maybe_compact` enforces token budget constraints
- **Reasoning folding** collapses any `REASONING` messages into the preceding `TOOL_CALL` payload to maintain conversation coherence
- **Serialization** transforms messages into backend-specific API formats through `fold_and_serialize`
- **Lock-based serialization** for single-GPU backends is handled transparently by the proxy layer

## Guardrail Implementation Details

While awaiting backend responses, the proxy applies a comprehensive guardrail stack that corrects errors and enforces safety constraints automatically.

### Response Validation and Rescue Parsing

The `ResponseValidator.validate` method in [[`src/forge/guardrails/response_validator.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/guardrails/response_validator.py)](https://github.com/antoinezambelli/forge/blob/main/src/forge/guardrails/response_validator.py) implements **rescue parsing** through `rescue_tool_call`. When a model accidentally emits tool-call JSON as plain text, this guardrail parses the malformed output and converts it back into a valid `ToolCall` object.

The same validator performs an **unknown-tool check**, detecting when the model generates tool names not advertised in the original request. When this occurs, the validator injects an *unknown-tool nudge*—a corrective user-role message that prompts the model to retry with a valid tool call.

### Error Budgeting and Retry Logic

Consecutive retry attempts are tracked by `ErrorTracker` in [[`src/forge/guardrails/error_tracker.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/guardrails/error_tracker.py)](https://github.com/antoinezambelli/forge/blob/main/src/forge/guardrails/error_tracker.py). This component maintains an error budget that stops the retry loop after a configurable number of failed attempts, raising `ToolCallError` to prevent infinite loops.

When the model returns bare text or invalid tool calls, the proxy automatically appends corrective nudges and retries the inference loop internally. Clients receive only the final validated response, never seeing the intermediate retry attempts.

### Context Compaction and Token Management

Before each backend call, `ContextManager` applies tiered trimming strategies to keep conversation histories within token budgets. This guardrail runs silently in the background, removing older messages when necessary while preserving critical context, ensuring the proxy never sends oversized requests to the backend.

### Synthetic Tool Injection and Stripping

The synthetic `respond` tool, defined in [[`src/forge/tools/respond.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/tools/respond.py)](https://github.com/antoinezambelli/forge/blob/main/src/forge/tools/respond.py), forces the model into structured tool-calling mode even for conversational responses. When the model invokes `respond(message="…")`, the proxy strips this internal tool call before formatting the output, converting it into standard text content that matches OpenAI's expected response schema.

## Transparent API Compatibility

After validation succeeds, the handler removes injected `respond` calls and formats the remaining tool calls or plain text back to OpenAI's response schema using `tool_calls_to_openai` or `text_response_to_openai` (and their SSE equivalents for streaming).

From the client's perspective, the proxy behaves identically to a standard OpenAI server. The request is sent once, and the response is either a single JSON object or a streaming SSE feed. All retry loops, nudge injections, context trimming, and tool stripping happen inside the proxy before any data reaches the client wire format.

## Practical Examples

### Starting the Proxy Server

You can launch the proxy in managed mode, which automatically handles backend lifecycle:

```python
from forge.proxy import ProxyServer

# Starts a local Llama.cpp backend on :8080 and the proxy on :8081

proxy = ProxyServer(backend="llamaserver", gguf="model.gguf")
proxy.start()           # blocks until the proxy is ready

print(f"Proxy listening at {proxy.url}")

# → http://127.0.0.1:8081

```

*Source*: [[`src/forge/proxy/proxy.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/proxy.py)](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/proxy.py)

### Making OpenAI-Compatible Requests with curl

Send standard OpenAI API requests to the proxy endpoint:

```bash
curl http://127.0.0.1:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "my-8b",
        "messages": [{"role":"user","content":"What is the weather in London?"}],
        "tools": [{"type":"function","function":{"name":"get_weather","description":"Get weather","parameters":{"type":"object","properties":{"city":{"type":"string"}}}}}]
      }'

```

If the model answers via the injected `respond` tool, the proxy strips it and returns clean JSON:

```json
{
  "id": "...",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {"role":"assistant","content":"The weather in London is sunny."},
      "finish_reason": "stop"
    }
  ]
}

```

### Using the OpenAI Python SDK

Existing OpenAI client code works without modification:

```python
import openai

client = openai.OpenAI(base_url="http://127.0.0.1:8081/v1")

resp = client.chat.completions.create(
    model="my-8b",
    messages=[{"role":"user","content":"Summarize the latest news"}],
    tools=[{
        "type":"function",
        "function": {
            "name":"summarize",
            "description":"Summarize a text",
            "parameters":{"type":"object","properties":{"text":{"type":"string"}}}
        }
    }]
)

print(resp.choices[0].message.content)

```

The SDK sees a standard OpenAI endpoint while the proxy handles all guardrails internally.

## Summary

- The **Forge proxy** exposes an OpenAI-compatible HTTP interface while running Forge's front-half inference pipeline internally.
- **Guardrails execute transparently** through `handle_chat_completions` and `run_inference`, applying context compaction, reasoning folding, and rescue parsing without client exposure.
- **Response validation** in [`response_validator.py`](https://github.com/antoinezambelli/forge/blob/main/response_validator.py) corrects malformed tool calls and detects unknown tools, injecting retry nudges automatically.
- **Error budgeting** via `ErrorTracker` prevents infinite retry loops by enforcing a configurable failure threshold.
- **Synthetic `respond` tool** injection ensures consistent tool-calling mode, with the tool stripped from final responses to maintain API compatibility.
- All preprocessing, validation, and retry logic occurs server-side, delivering only clean standard responses to OpenAI SDK clients.

## Frequently Asked Questions

### What guardrails does the Forge proxy apply automatically?

The proxy applies **context compaction** to enforce token limits, **rescue parsing** to fix malformed JSON tool calls, **unknown-tool detection** to correct hallucinated tool names, and **retry nudges** to recover from invalid outputs. These run inside `run_inference` and `ResponseValidator.validate` before any response reaches the client.

### How does the proxy handle malformed tool calls from the backend?

When a model emits tool-call JSON as plain text or formatted incorrectly, the `rescue_tool_call` function in [`response_validator.py`](https://github.com/antoinezambelli/forge/blob/main/response_validator.py) parses the output and reconstructs a valid `ToolCall` object. If the tool name doesn't match the request schema, the proxy injects a corrective user message and retries the inference loop automatically.

### Why is the synthetic `respond` tool necessary?

The `respond` tool, defined in [`respond.py`](https://github.com/antoinezambelli/forge/blob/main/respond.py), forces the model into structured tool-calling mode even when generating conversational text. This maintains consistency in the inference pipeline, allowing the same validation logic to handle both functional calls and plain responses. The proxy strips these synthetic calls before sending the final output to the client.

### Can I use the proxy with existing OpenAI client libraries?

Yes. The proxy implements the standard `POST /v1/chat/completions` endpoint and returns OpenAI-compatible JSON or SSE streams. You can point any OpenAI SDK client to the proxy's base URL (`http://127.0.0.1:8081/v1`), and all guardrail processing—retry loops, context trimming, and tool injection—happen transparently without requiring client-side code changes.