# How TokenJuice Compresses Tool Output Before It Reaches the Model

> Learn how TokenJuice compresses tool output with a two stage pipeline using ML and Python to replace text with token identifiers, preserving correctness and minimizing token usage.

- Repository: [Tiny Humans/openhuman](https://github.com/tinyhumansai/openhuman)
- Tags: deep-dive
- Published: 2026-08-24

---

**TokenJuice implements a two-stage compression pipeline that uses a host-side ML service and Python side-car to shrink tool output, replacing lengthy text with short token identifiers that preserve correctness while minimizing token usage.**

TokenJuice is an advanced compression system in the `tinyhumansai/openhuman` repository designed to reduce context window pressure when large tool outputs are sent to language models. Unlike simple truncation, TokenJuice employs an intelligent ML-based compressor that maintains semantic integrity while dramatically reducing token count. This article explains exactly how TokenJuice compresses tool output before it reaches the model, covering the host-side service architecture, Python side-car implementation, and token-based caching mechanism.

## The Two-Stage Compression Architecture

The compression process operates through a strict host-side delegation pattern that prevents model modules from bypassing the compression contract.

### Stage 1: Host-Side ML Service Registration

The core process registers a **TinyBus** service named `MlHost` in [`src/openhuman/modules/tokenjuice_host.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/modules/tokenjuice_host.rs). This service exposes a `compress` method that delegates execution to `crate::openhuman::inference::tokenjuice::ml::compress`. By routing compression through the host rather than allowing direct module access, the system ensures that any implementation drift causes a graceful fallback to uncompressed output, preserving correctness.

### Stage 2: Python Side-Car Execution

The actual compression logic runs inside a lightweight **Python side-car** launched by the runtime. Located in the ML implementation layer ([`src/openhuman/inference/tokenjuice/ml.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/inference/tokenjuice/ml.rs)), this side-car loads a small transformer model configured via `tokenjuice.ml_*` settings. The `compress` async function transmits the raw tool output to the side-car, which returns a shortened string that retains essential information while using significantly fewer tokens.

## Token-Based Caching and Retrieval

After compression, the system does not immediately send the compressed text to the model. Instead, it employs a token-exchange mechanism that minimizes active context usage.

### Cache Storage and Identifier Generation

The compressed string is stored in the **TokenJuice cache** ([`src/openhuman/inference/tokenjuice/cache.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/inference/tokenjuice/cache.rs)). The cache returns a short token identifier, which is wrapped in a JSON structure like `{"token":"abc123"}` and passed to the model. This approach ensures the model receives only a minimal token footprint regardless of the original output size.

### Lazy Retrieval via Tool Call

When the model requires the full content, it invokes the `tokenjuice_retrieve` tool implemented in [`src/openhuman/inference/tokenjuice/tools.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/inference/tokenjuice/tools.rs). This tool accepts the token identifier and returns the original uncompressed payload from the cache, enabling efficient random access to large tool outputs without bloating the active context window.

## Configuration Schema and Tuning

Compression behavior is controlled through structured configuration defined in [`src/openhuman/config/schema/tokenjuice.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/config/schema/tokenjuice.rs).

Key parameters include:

- `ml_compression_enabled`: Boolean toggle to activate compression
- `ml_model_id`: Identifier for the transformer model (e.g., "tinyjuice-compressor-v1")
- `ml_device`: Compute target ("cpu" or "cuda")
- `ml_target_ratio`: Target compression ratio (e.g., 0.3 for 30% of original size)
- `ml_max_input_chars`: Maximum characters to process

Example configuration:

```rust
// Config structure in schema/tokenjuice.rs
{
    "tokenjuice": {
        "ml_compression_enabled": true,
        "ml_model_id": "tinyjuice-compressor-v1",
        "ml_device": "cpu",
        "ml_target_ratio": 0.3,
        "ml_max_input_chars": 8000
    }
}

```

## Implementation Flow and Fallback Behavior

The compression integration follows a defensive programming pattern that prioritizes reliability over compression ratio.

When a tool finishes execution, the core checks `config.tokenjuice.ml_compression_enabled`. If enabled, it calls `tokenjuice::ml::compress(&raw_output, &options).await`. The function returns `Option<String>`—`Some(compact)` on success or `None` on failure, triggering an automatic fallback to the raw uncompressed output.

```rust
// Core compression logic in the tool execution path
let raw_output = tool.run(...);
let compressed = if config.tokenjuice.ml_compression_enabled {
    // Host-side TinyBus call delegates to Python side-car
    tokenjuice::ml::compress(&raw_output, &options).await?
} else {
    None
};

match compressed {
    Some(compact) => {
        // Store compressed content and return token identifier
        let token = tokenjuice::cache::store(compact);
        format!("{{\"token\":\"{token}\"}}")
    }
    None => raw_output, // Fallback: send uncompressed text
}

```

When the model later needs the original content, it uses the retrieval tool:

```rust
// Retrieval from cache via tokenjuice_retrieve tool
use tokenjuice::retrieve;

let token = "abc123"; // Token received from model
let original = retrieve(token.to_string(), None).await?;
println!("Original output: {}", original);

```

## Summary

- **TokenJuice** uses a host-side **TinyBus** service (`MlHost`) in [`src/openhuman/modules/tokenjuice_host.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/modules/tokenjuice_host.rs) to delegate compression, preventing modules from bypassing the system.
- A **Python side-car** performs ML-based compression via [`src/openhuman/inference/tokenjuice/ml.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/inference/tokenjuice/ml.rs), using configurable models specified in [`src/openhuman/config/schema/tokenjuice.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/config/schema/tokenjuice.rs).
- The system stores compressed content in a dedicated cache ([`src/openhuman/inference/tokenjuice/cache.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/inference/tokenjuice/cache.rs)) and returns short token identifiers to the model, minimizing context window usage.
- The `tokenjuice_retrieve` tool in [`src/openhuman/inference/tokenjuice/tools.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/inference/tokenjuice/tools.rs) enables lazy loading of full payloads when the model specifically requests them.
- Built-in fallback mechanisms ensure that compression failures result in uncompressed delivery rather than errors, maintaining system reliability.

## Frequently Asked Questions

### What happens if the TokenJuice compression fails?

If the `compress` function returns `None` or the host service is unavailable, the system automatically falls back to sending the raw uncompressed tool output to the model. This ensures reliability while attempting compression on a best-effort basis, as implemented in the match logic of the core tool execution flow.

### Where is the compressed data stored before retrieval?

The compressed strings are stored in the **TokenJuice cache** managed in [`src/openhuman/inference/tokenjuice/cache.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/inference/tokenjuice/cache.rs). The cache maps short token identifiers to their full payloads, allowing the model to reference large outputs without holding them in the active context window, effectively moving storage from the context to a side cache.

### How do I enable or configure TokenJuice compression?

Enable compression by setting `ml_compression_enabled: true` in the configuration schema located at [`src/openhuman/config/schema/tokenjuice.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/config/schema/tokenjuice.rs). You can tune the `ml_target_ratio` to control compression aggressiveness and specify the `ml_model_id` to select which transformer model the Python side-car loads, with options for device targeting via `ml_device`.

### Why does TokenJuice use a host-side service instead of module-level compression?

The host-side **TinyBus** architecture in [`tokenjuice_host.rs`](https://github.com/tinyhumansai/openhuman/blob/main/tokenjuice_host.rs) enforces a strict contract between the core system and compression implementation according to the `tinyhumansai/openhuman` source code. If a module attempts to drift from this contract or the compression service is unavailable, the module simply receives uncompressed data, preventing silent failures or inconsistent behavior while maintaining security boundaries.