# Caveman Source Map Explained: How Engine, Proxy, MCP, and Browser Modules Interact

> Understand the Caveman source map. Learn how Engine, Proxy, MCP, and Browser modules interact to map output tokens to original source locations for deterministic execution and replay.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: internals
- Published: 2026-09-04

---

**The Caveman source map is a JSON metadata structure generated by the Engine that maps every output token to its original source location, enabling the Proxy to persist deterministic execution contexts and the MCP recovery tool to replay exact tool sequences across all modules.**

The **JuliusBrussee/caveman** repository implements a runtime architecture for LLM-driven agents centered on observable, reproducible execution. The **source map** serves as the backbone of this system, tracking the provenance of every token and tool call to enable deterministic replay and compression. This article examines how the source map functions and how the **Engine**, **Proxy**, **MCP**, and **Browser** modules interact to maintain execution integrity.

## What Is the Caveman Source Map?

The source map is a JSON object produced by the **Engine** during prompt rewriting and model inference. It records the exact origin—file and line number—of every token and chunk in the generated output, enabling **deterministic replay** and **live-zone compression** for subscription agents.

In [`rewriter/gate.go`](https://github.com/JuliusBrussee/caveman/blob/main/rewriter/gate.go), the function `sourceLocations(text string) []string` extracts source references using regular expressions that recognize `file:line` patterns. The resulting structure attaches to the Engine's JSON response:

```json
{
  "tokens": [
    {"text":"Hello", "src":"prompt:1"},
    {"text":"world", "src":"prompt:1"},
    {"text":"[TOOL]", "src":"tool:my_search:3"}
  ],
  "chunks": [
    {"id":"tool:my_search", "src":"tool:my_search:3"}
  ]
}

```

Each `src` field represents a source location (e.g., `prompt:1` for line 1 of the user prompt, `tool:my_search:3` for line 3 of a tool definition). This mapping allows the system to substitute compressed tokens without losing traceability to the original source.

## The Four Core Modules and Their Roles

### Engine (The Compute Layer)

The **Engine** is a compiled Go binary (`caveman-engine`) that performs the heavy lifting: prompt rewriting, token counting, compression, and streaming responses from LLM providers like OpenAI and Anthropic. According to the source analysis, the Engine is the only component that contacts underlying providers. It returns data via JSON-over-STDIO, including the generated source map constructed in [`rewriter/gate.go`](https://github.com/JuliusBrussee/caveman/blob/main/rewriter/gate.go) and serialized in [`engine/output.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/output.go).

### Proxy (The Orchestration Layer)

The **Proxy** (`caveman-proxy`) is a Go service defined in [`proxy/internal/gateway/proxy.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/internal/gateway/proxy.go) that sits between agents and the Engine. It handles three critical functions: **MCP tool schema sharing**, **run-state publishing**, and **token budgeting**. The Proxy writes a JSON run-state file (`caveman.proxy.run.v1`) that records active MCP servers and recovery flags, and it aggregates per-turn token usage into a SQLite store. When the Engine returns a response, the Proxy extracts the source map and persists it to the **CCR store** (Context-Compression-Recovery store).

### MCP (The Recovery Protocol)

The **MCP** (Model-Context-Protocol) integration enables deterministic recovery via the `caveman_retrieve` tool. When **RecoveryViaMCP** is enabled (recorded in [`proxy/internal/runstate/runstate.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/internal/runstate/runstate.go)), the Proxy omits the Engine's server-side recovery loop and instead relies on the stored source map to reconstruct exact tool sequences. Agents invoke `caveman_retrieve` to fetch previous execution contexts, and the Proxy uses the persisted map to guarantee byte-stable replay.

### Browser (The Client Extension)

The **Browser** module resides in the optional MV3 extension (`caveman-browser`), specifically in [`extension/src/directive.js`](https://github.com/JuliusBrussee/caveman/blob/main/extension/src/directive.js). It implements client-side browsing workflows including page fetching, DOM parsing, and user-action simulation. The extension communicates with the Proxy via the local HTTP gateway (`CAVEMAN_GATEWAY_URL`). When executing browse actions, the Engine generates source maps for generated actions, allowing the extension to highlight the original script lines that triggered specific clicks or navigation events.

## How the Modules Interact via the Source Map

The interaction follows a strict pipeline that ensures every token remains traceable:

1. **Initialization**: The CLI spawns the Proxy, which writes the run-state file declaring active MCP servers and whether recovery via MCP is enabled, as implemented in [`proxy/internal/runstate/runstate.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/internal/runstate/runstate.go).

2. **Request Routing**: When an agent issues a request (standard LLM call or browser action), the Proxy looks up the appropriate provider mount in `proxy/providers/*.go` and forwards the payload to the Engine binary.

3. **Engine Processing**: The Engine rewrites the prompt, executes the model, and returns both the content and the source map. The map generation logic resides in [`rewriter/gate.go`](https://github.com/JuliusBrussee/caveman/blob/main/rewriter/gate.go), where `sourceLocations` extracts file references.

4. **Storage and Decoration**: The Proxy receives the Engine's response, records token usage in SQLite, and stores the source map in the CCR store. If `RecoveryViaMCP` is active, the Proxy flags the response as recoverable through the MCP tool rather than internal Engine state.

5. **Deterministic Recovery**: When the agent later calls `caveman_retrieve`, the Proxy queries the stored source map to replay the exact sequence of tool calls, ensuring deterministic reconstruction even if the Engine process was restarted.

## Implementation: Generating and Storing Source Maps

The generation logic in [`rewriter/gate.go`](https://github.com/JuliusBrussee/caveman/blob/main/rewriter/gate.go) uses regular expressions to parse transformed prompts. The Engine emits the map as follows:

```go
// rewriter/gate.go - source extraction
func sourceLocations(text string) []string {
    // Regex-based extraction of file:line patterns
    // Returns slice of source references like ["prompt:1", "tool:search:3"]
}

// Engine response construction
srcMap := map[string]any{
    "tokens": []map[string]string{
        {"text": token, "src": fmt.Sprintf("%s:%d", srcFile, srcLine)},
    },
}
json.NewEncoder(os.Stdout).Encode(map[string]any{
    "content":   finalText,
    "sourceMap": srcMap,
})

```

The Proxy then persists this data alongside execution metadata:

```go
// proxy/internal/gateway/proxy.go
var engineResp struct {
    Content   string          `json:"content"`
    SourceMap json.RawMessage `json:"sourceMap"`
}
json.NewDecoder(resp.Body).Decode(&engineResp)

// Persist with token stats in CCR store
store.SaveSourceMap(runID, engineResp.SourceMap)

```

## Practical Example: MCP Recovery Flow

Agents interact with the system through the Python SDK or direct HTTP calls. To recover a previous execution state deterministically:

```python

# packages/sdk/python/client.py usage pattern

result = client.run_tool(
    name="caveman_retrieve",
    args={"run_id": current_run_id}
)

# Returns exact tool sequence reconstructed from the source map

print(result["replay"])

```

The Proxy uses the stored source map from the CCR store to ensure the replay matches the original execution byte-for-byte, preserving workflow integrity across the Engine, Proxy, and Browser boundaries.

## Summary

- The **source map** is a JSON structure generated in [`rewriter/gate.go`](https://github.com/JuliusBrussee/caveman/blob/main/rewriter/gate.go) that maps every output token to its original `file:line` source location using the `sourceLocations` function.
- The **Engine** produces the source map during prompt rewriting and returns it via JSON-over-STDIO alongside model responses.
- The **Proxy** persists the source map in the CCR store (SQLite) and manages token budgeting and run-state in [`proxy/internal/runstate/runstate.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/internal/runstate/runstate.go).
- **MCP** integration via `caveman_retrieve` enables agents to replay exact execution contexts using stored source maps instead of Engine-side recovery loops.
- The **Browser** extension ([`extension/src/directive.js`](https://github.com/JuliusBrussee/caveman/blob/main/extension/src/directive.js)) communicates through the Proxy to receive source-mapped browsing actions for observable client-side automation.

## Frequently Asked Questions

### What format does the Caveman source map use?

The source map uses a JSON format containing `tokens` and `chunks` arrays. Each entry includes a `src` field with `file:line` references (e.g., `prompt:1` or `tool:my_search:3`) extracted by the `sourceLocations` function in [`rewriter/gate.go`](https://github.com/JuliusBrussee/caveman/blob/main/rewriter/gate.go). This structure enables the Proxy to correlate compressed output with original source positions.

### How does the Proxy store source map data?

The Proxy persists source maps in the **CCR store**, typically implemented as a SQLite database alongside token usage statistics. This occurs in [`proxy/internal/gateway/proxy.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/internal/gateway/proxy.go) immediately after receiving the Engine's JSON response, ensuring the map is available for later MCP recovery operations.

### Can the Browser extension access source maps directly?

No, the Browser extension ([`extension/src/directive.js`](https://github.com/JuliusBrussee/caveman/blob/main/extension/src/directive.js)) does not access source maps directly. It communicates with the Proxy via the `CAVEMAN_GATEWAY_URL` HTTP endpoint. The Browser receives source-mapped actions indirectly when the Proxy forwards Engine responses containing browsing metadata and source location headers.

### Which file generates the source locations in Caveman?

The [`rewriter/gate.go`](https://github.com/JuliusBrussee/caveman/blob/main/rewriter/gate.go) file contains the core generation logic, specifically the `sourceLocations(text string) []string` function. This function applies regular expressions to transformed prompts to identify `file:line` patterns, which the Engine then assembles into the final source map attached to its output.