Caveman Engine Stable Operations: The Five Core API Methods Explained

The JuliusBrussee/caveman repository exposes five stable operations—Compress, Retrieve, Detect, Stats, and Simulate—that form the immutable contract between the engine core and its consumers.

According to the Caveman Engine source code, these methods constitute the four-call surface (plus one auxiliary dry-run method) that remains compatible across releases. Whether you are building a proxy server, CLI tool, or MCP integration, these functions in the engine package provide the only touchpoints you need.

The Stable Contract

The engine intentionally limits its public API to ensure long-term stability. As documented in the file header of engine/engine.go, the surface area includes four primary calls—Compress, Retrieve, Detect, and Stats—plus an auxiliary Simulate method for dry-run testing. These methods are implemented in the main engine file and backed by the CCR store and compressor registry.

Compress

The Compress method detects content type, routes payloads to the appropriate compressor, and returns compressed bytes alongside accounting metadata. It operates on a fail-closed principle: if no compressor matches, if compression does not reduce size, or if a lossy transform cannot be recovered, the original bytes return unchanged.

Source: engine/engine.go#L63

func exampleCompress(e *engine.Engine, data []byte) {
    opts := engine.Options{Mode: engine.ModeCompress}
    result, err := e.Compress(data, opts)
    if err != nil {
        log.Fatalf("compress error: %v", err)
    }
    fmt.Printf("Compressed %d → %d bytes (ratio %.2f)\n",
        result.TokensBefore, result.TokensAfter, result.Ratio)
    if result.RecoveryHandle != "" {
        fmt.Printf("Recovery handle: %s\n", result.RecoveryHandle)
    }
}

Retrieve

The Retrieve method accepts a recovery handle generated by a lossy compressor and returns the exact original bytes from the CCR store. If the handle does not exist in the store, it returns ccr.ErrNotFound.

Source: engine/engine.go#L47

func exampleRetrieve(e *engine.Engine, handle string) {
    original, err := e.Retrieve(handle)
    if err != nil {
        log.Fatalf("retrieve error: %v", err)
    }
    fmt.Printf("Recovered %d original bytes\n", len(original))
}

Detect

The Detect operation inspects raw payloads to infer content type when the caller omits explicit typing. This inference feeds directly into the routing decisions made by Compress and Simulate, ensuring optimal compressor selection without manual configuration.

Stats

The Stats method aggregates compression accounting data from the CCR store, reporting token savings, per-content-type buckets, and operation counts. When no store is configured, it returns an empty stats struct rather than an error.

Source: engine/engine.go#L78

func exampleStats(e *engine.Engine) {
    stats, err := e.Stats()
    if err != nil {
        log.Fatalf("stats error: %v", err)
    }
    fmt.Printf("Total tokens saved: %d\n", stats.TotalTokensSaved)
    for ct, bucket := range stats.ByContentType {
        fmt.Printf("- %s: %d saves (%d compressions)\n",
            ct, bucket.TokensSaved, bucket.Count)
    }
}

Simulate

The Simulate method executes the full detection, routing, and compression pipeline without persisting state. It performs no CCR Put operations and issues no network calls, allowing you to preview token reduction and verify recoverability before committing storage.

Source: engine/engine.go#L42

func exampleSimulate(e *engine.Engine, data []byte) {
    opts := engine.Options{Mode: engine.ModeCompress}
    sim := e.Simulate(data, opts)
    fmt.Printf("Simulation – would save %d tokens (ratio %.2f)\n",
        sim.TokensSaved, sim.Ratio)
    fmt.Printf("Recoverable: %v, Lossy: %v\n", sim.Recoverable, sim.Lossy)
}

Implementation Architecture

The stable operations rely on three internal components that are not part of the public contract but power the methods above:

• engine/engine.go – Contains the method implementations for the five stable operations and maintains the engine state.
• engine/ccr/store.go – Implements the CCR (Content Compression and Recovery) store that backs Retrieve and Stats with id spaces and object tables.
• engine/compressors/registry.go – Maintains the compressor registry used by Compress and Simulate to route payloads based on detected content types.

These files are located in the engine/ directory of the JuliusBrussee/caveman repository.

Summary

• Compress, Retrieve, Detect, and Stats form the four-call stable contract defined in engine/engine.go.
• Simulate provides a dry-run variant of Compress for testing token savings without side effects.
• All five methods are implemented in engine/engine.go at specific line offsets (L42, L47, L63, L78).
• The engine is fail-closed: unsuccessful compression or missing recovery handles return original data or sentinel errors rather than panic.
• External consumers—proxy servers, SDKs, and CLI tools—should interact exclusively through these five operations to ensure forward compatibility.

Frequently Asked Questions

What are the stable operations exposed by the Caveman Engine?

The Caveman Engine exposes five stable operations: Compress, Retrieve, Detect, Stats, and Simulate. The first four constitute the core four-call surface, while Simulate acts as a dry-run auxiliary. These are implemented in engine/engine.go and remain constant across releases.

How does Simulate differ from Compress?

Simulate mirrors the exact logic of Compress—including detection, routing, and compression—but skips persistence. It does not write to the CCR store and issues no network calls, making it ideal for previewing token savings. In contrast, Compress persists recovery handles and accounts for statistics.

Where is the Detect function implemented?

The Detect operation is part of the stable contract declared in the header comments of engine/engine.go. While the analysis Localizer indicates its definition resides within the engine package, it is invoked internally by both Compress and Simulate when callers omit explicit content-type metadata.

What happens if Retrieve cannot find a recovery handle?

If the handle does not exist in the CCR store, Retrieve returns ccr.ErrNotFound. The method guarantees exact original byte recovery when the handle is present, but signals failure explicitly rather than returning partial or corrupted data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}