# How to Orchestrate Sequential Compression Transforms in Headroom's TransformPipeline

> Learn how Headroom orchestrates sequential compression transforms in its TransformPipeline. Discover three-stage logic for reformatting, bloat estimation, and offload application to optimize your data processing.

- Repository: [Tejas Chopra/headroom](https://github.com/chopratejas/headroom)
- Tags: how-to-guide
- Published: 2026-06-13

---

**Headroom orchestrates sequential compression transforms through a three-stage pipeline that applies reformat transforms sequentially with early-exit logic, estimates bloat in parallel for offload candidates, and conditionally applies offloads based on configurable thresholds.**

Headroom is a Rust-based compression proxy that processes HTTP payloads through a sophisticated `TransformPipeline`. Understanding how to orchestrate sequential compression transforms within this architecture is essential for building efficient compression workflows that balance compute cost against payload reduction. The pipeline implementation resides primarily in `chopratejas/headroom`, specifically within the orchestrator module.

## The Three-Stage Pipeline Architecture

Headroom processes every request through a precisely ordered execution flow defined in `CompressionPipeline::run` (lines 1004‑1097 of [`crates/headroom-core/src/transforms/pipeline/orchestrator.rs`](https://github.com/chopratejas/headroom/blob/main/crates/headroom-core/src/transforms/pipeline/orchestrator.rs)). This architecture separates lightweight reformatting from compute-intensive offloading through distinct phases.

### Sequential Reformat Phase

The pipeline begins by applying **reformat transforms** strictly sequentially in their registration order. As implemented in `CompressionPipeline::run_reformats` (lines 1010‑1024), each transform mutates the content before passing it to the next stage.

After every reformat operation, the pipeline calculates a **reformat target ratio** (`output_len / original_len`). If this ratio falls below the configured threshold, the remaining reformat transforms are immediately skipped. This early-exit optimization prevents unnecessary processing when the payload has already reached satisfactory compression levels.

### Parallel Bloat Estimation Phase

Once reformatting completes, the pipeline evaluates potential offload transforms. Unlike the sequential reformat phase, bloat estimation runs **in parallel** using Rayon’s `par_iter().map(|o| o.estimate_bloat(content))` pattern (lines 1057‑1064). 

Each offload transform implements a bloat estimator that calculates how many bytes the transform could potentially remove from the current payload. This parallel assessment ensures that expensive estimation logic does not block the pipeline, while the results determine which offloads actually execute.

### Gated Sequential Offload Phase

Accepted offloads are applied **sequentially**, with each transform receiving the output of the previous one. The decision logic (starting at line 1039) compares each offload’s estimated score against two configurable gates:

- **`bloat_threshold`** – Runs the offload unconditionally if the estimated byte reduction exceeds this value
- **`offload_fallback_ratio`** – Triggers the offload when the reformat phase failed to shrink the payload sufficiently *and* the bloat estimate is positive

This gating mechanism ensures that expensive compression operations only execute when they provide measurable benefit.

## Configuration and Thresholds

The pipeline’s behavior is controlled through `PipelineConfig`, defined in [`crates/headroom-core/src/transforms/pipeline/config.rs`](https://github.com/chopratejas/headroom/blob/main/crates/headroom-core/src/transforms/pipeline/config.rs) (lines 1001‑1080). This structure exposes the ratios and thresholds that govern the early-exit and gating logic.

Configuration options include the reformat target ratio, bloat thresholds, and per-domain bloat-estimator parameters. You can supply custom configurations via `PipelineConfig::default()` or load them from TOML definitions located in [`config/pipeline.toml`](https://github.com/chopratejas/headroom/blob/main/config/pipeline.toml).

## Practical Implementation Example

The following example demonstrates how to build and execute a pipeline with sequential reformat and gated offload transforms:

```rust
use headroom_core::{
    CompressionPipeline, PipelineConfig, ContentType,
    transforms::{
        pipeline::reformats::JsonMinifier,
        pipeline::offloads::JsonOffload,
    },
};
use headroom_core::ccr::InMemoryCcrStore;
use headroom_core::transforms::pipeline::CompressionContext;

// 1️⃣ Build a pipeline -------------------------------------------------------
let pipeline = CompressionPipeline::builder()
    // Register reformat transforms – they run sequentially.
    .with_reformat(JsonMinifier)               // → strips whitespace from JSON
    // Register offload transforms – they are evaluated in parallel,
    // then applied sequentially if the gates pass.
    .with_offload(JsonOffload)                 // → SmartCrusher JSON offload
    // Optional: supply a custom config (or rely on defaults).
    .with_config(PipelineConfig::default())
    .build();

// 2️⃣ Prepare runtime context ------------------------------------------------
let ctx = CompressionContext::default();       // per‑request context
let store = InMemoryCcrStore::new();          // CCR cache (used by offloads)

// 3️⃣ Run the pipeline --------------------------------------------------------
let input = r#"
{
  "a": 1,
  "b": 2,
  "c":   3
}
"#;                                            // pretty‑printed JSON

let result = pipeline.run(
    input,
    ContentType::JsonArray,                  // content type detection
    &ctx,
    &store,
);

// 4️⃣ Inspect the result ------------------------------------------------------
println!("Final output ({} bytes)", result.output.len());
println!("Saved bytes: {}", result.bytes_saved);
println!("Applied steps: {:?}", result.steps_applied);
println!("CCR keys: {:?}", result.cache_keys);

```

**Key registration points:**

- **`with_reformat`** (lines 1086‑1098): Adds transforms to the sequential reformat queue
- **`with_offload`** (lines 1101‑1113): Registers offloads for parallel bloat estimation and conditional sequential execution
- **`run`**: Executes the complete three-stage flow

## Summary

- **Reformat transforms** execute sequentially with early-exit logic based on the reformat target ratio, implemented in `CompressionPipeline::run_reformats`
- **Bloat estimation** runs in parallel using Rayon to evaluate offload candidates without blocking the pipeline
- **Gated offloads** apply sequentially only when estimated savings exceed `bloat_threshold` or when reformat results fall below `offload_fallback_ratio`
- **Configuration** is centralized in `PipelineConfig`, allowing per-deployment tuning of compression aggressiveness
- The **builder pattern** (`CompressionPipeline::builder()`) enables declarative composition of complex transform chains

## Frequently Asked Questions

### What is the difference between reformat and offload transforms in Headroom?

**Reformat transforms** are lightweight operations that modify content representation (such as `JsonMinifier` stripping whitespace) and run sequentially until a size threshold is met. **Offload transforms** are compute-intensive operations (like `JsonOffload`) that estimate their potential impact in parallel, then execute sequentially only if their bloat scores pass configured gates. As defined in [`crates/headroom-core/src/transforms/pipeline/traits.rs`](https://github.com/chopratejas/headroom/blob/main/crates/headroom-core/src/transforms/pipeline/traits.rs), these implement distinct traits: `ReformatTransform` versus `OffloadTransform`.

### How does the pipeline decide when to skip remaining transforms?

After each reformat transform completes, `CompressionPipeline::run_reformats` calculates the ratio of current output length to original length. If this ratio drops below the configured reformat target ratio, the function immediately returns, skipping all remaining reformat transforms. This early-exit optimization prevents wasted CPU cycles on already-compressed payloads.

### Can I configure custom thresholds for the gated offload phase?

Yes. The `PipelineConfig` structure exposes `bloat_threshold` and `offload_fallback_ratio` parameters. The `bloat_threshold` sets a minimum byte-reduction estimate for unconditional offload execution, while `offload_fallback_ratio` triggers offloads when reformatting underperforms but positive bloat remains. These are deserialized from [`config/pipeline.toml`](https://github.com/chopratejas/headroom/blob/main/config/pipeline.toml) or set programmatically via the builder’s `with_config` method.

### Where is the orchestration logic implemented in the source code?

The core orchestration resides in [`crates/headroom-core/src/transforms/pipeline/orchestrator.rs`](https://github.com/chopratejas/headroom/blob/main/crates/headroom-core/src/transforms/pipeline/orchestrator.rs). Specifically, `CompressionPipeline::run` (lines 1004‑1097) coordinates the three-phase execution, while `run_reformats` handles sequential reformatting (lines 1010‑1024) and `estimate_bloats` manages parallel estimation (lines 1057‑1064). Trait definitions live in [`traits.rs`](https://github.com/chopratejas/headroom/blob/main/traits.rs), and the provided transforms are implemented in [`reformats/json_minifier.rs`](https://github.com/chopratejas/headroom/blob/main/reformats/json_minifier.rs) and [`offloads/json_offload.rs`](https://github.com/chopratejas/headroom/blob/main/offloads/json_offload.rs).