Project Structure of Headroom: Understanding the Polyglot Repository Layout

Headroom organizes its Python CLI, Rust proxy core, and TypeScript SDK into modular directories, with the main Python package in headroom/, the high-performance proxy in crates/headroom-core/, and developer tooling in scripts/ and docker/.

The chopratejas/headroom repository is a mixed-language codebase that combines a Python command-line interface with a Rust-based proxy engine. Understanding the headroom project structure is essential for contributing to the transform pipeline, extending the proxy server, or integrating the TypeScript SDK.

Top-Level Directory Layout

The repository root contains twelve primary directories and configuration files that separate concerns by language and function:

• headroom/ – Primary Python package containing the CLI entry point and transform pipeline
• crates/headroom-core/ – Rust implementation of the proxy core and signal processing library
• sdk/ – Official TypeScript SDK packaged for npm
• benchmarks/ – Automated performance tests for transforms and proxy modes
• docker/ – Docker Compose files and bake scripts for local development
• wiki/ – Markdown documentation and troubleshooting guides
• examples/ – Sample applications including Vercel AI SDK and LangChain demos
• scripts/ – Shell and PowerShell helpers for installation and CI
• plugins/ – Optional extensions including OpenClaw, Hermes, and OAuth2
• tests/ – Pytest suite covering transforms and integration tests
• REALIGNMENT/ – Design decision documents and architectural roadmap

Key configuration files at the root include pyproject.toml for Python dependencies, Cargo.toml for the Rust workspace, and Dockerfile for containerization.

Python Package Architecture (headroom/)

The headroom/ directory implements the CLI and transform pipeline. According to the source code, this package contains several critical sub-modules:

CLI and Entry Points

The headroom/cli/ subdirectory houses the Click-based command-line interface. The headroom/cli/wrap.py file handles the headroom entry point, parsing arguments and dispatching to the appropriate handlers.

Transform Pipeline

The headroom/transforms/ directory contains the core processing logic. Each file implements a specific compression or analysis transform:

• log_compressor.py – Compresses log output
• search_compressor.py – Optimizes search queries
• tag_protector.py – Protects sensitive tags during processing

The headroom/transforms/pipeline.py module orchestrates these transforms sequentially, as shown in the usage example below.

Utility Modules

Supporting functionality resides in:

• headroom/utils.py – Generic helper functions for file I/O and path handling
• headroom/update_check.py – Auto-update logic contacting the upstream release feed

Rust Core Implementation (crates/headroom-core/)

The Rust components deliver high-performance signal processing and proxy capabilities. Located in crates/headroom-core/, this crate contains two primary domains:

Signal Processing

The crates/headroom-core/src/signals/ directory implements the core signal processing library. These modules handle the low-level transformations that process text streams before they reach the proxy server.

Proxy Server

The crates/headroom-core/src/proxy/ directory contains the proxy server implementation. This component receives incoming requests and forwards them through the transform pipeline, bridging the Rust core with the Python transforms.

SDK and Integration Examples

The repository provides first-party SDKs and sample implementations in dedicated directories:

TypeScript SDK

The sdk/ directory contains the official TypeScript SDK, distributed via npm. This package wraps the HTTP API exposed by the Rust proxy, allowing JavaScript applications to leverage Headroom's compression capabilities.

Sample Applications

The examples/ directory includes production-ready demonstrations:

• Vercel AI SDK integration patterns
• LangChain compatibility demos
• macOS launch-agent configurations

Development and Testing Infrastructure

Headroom separates operational code from development utilities:

Testing and Benchmarks

The tests/ directory contains the Pytest suite covering core transforms, memory queries, and CLI wrappers. Performance validation occurs in benchmarks/, where benchmarks/run_benchmarks.py serves as the entry point for automated relevance and throughput testing.

Containerization and Scripts

Local development relies on docker/docker-compose.yml to spin up the Rust proxy alongside the Python CLI. The scripts/ directory provides installation helpers and CI orchestration scripts for both Unix and Windows environments.

Working with the Codebase

The following examples demonstrate how the project structure translates to actual usage across the three languages.

Python Transform Pipeline

To invoke transforms from the Python package:

from headroom.transforms.pipeline import Pipeline

# Build a pipeline that compresses logs and then tags protected words

pipeline = Pipeline([
    "log_compressor",
    "tag_protector",
])

original = "User password is secret123"
compressed = pipeline.run(original)
print(compressed)

Rust Signal Processing

When extending the Rust core, you interact with the signal processor:

use headroom_core::signals::{Signal, Processor};

let mut proc = Processor::new();
proc.add_signal(Signal::new("log_compress"));
let output = proc.process("This is a long log line ...");
println!("{}", output);

TypeScript SDK Usage

To consume the proxy from TypeScript:

import { HeadroomClient } from "headroom-sdk";

const client = new HeadroomClient({ endpoint: "http://localhost:8000" });
const result = await client.compress("Long documentation text...");
console.log(result);

Summary

• Headroom organizes code by language: Python in headroom/, Rust in crates/headroom-core/, and TypeScript in sdk/.
• The transform pipeline resides in headroom/transforms/ and is orchestrated by headroom/transforms/pipeline.py.
• The Rust proxy core splits between crates/headroom-core/src/signals/ for processing and crates/headroom-core/src/proxy/ for the server.
• Configuration is managed via pyproject.toml for Python and Cargo.toml for Rust.
• Documentation lives in wiki/ and REALIGNMENT/, while examples/ provides integration patterns.

Frequently Asked Questions

What is the purpose of the headroom/transforms/ directory?

The headroom/transforms/ directory houses the core compression and analysis logic of the Python package. Each module, such as log_compressor.py and tag_protector.py, implements a specific transformation that can be chained together via headroom/transforms/pipeline.py to process text before it reaches the proxy.

How does the Rust proxy integrate with the Python CLI?

The Rust proxy in crates/headroom-core/src/proxy/ runs as a standalone server that receives HTTP requests. The Python CLI can communicate with this proxy to offload high-performance signal processing tasks, while the Python layer handles the transform pipeline orchestration and user interface.

Where are the build configurations defined?

Python dependencies and build settings are defined in pyproject.toml at the repository root, while Rust uses crates/headroom-core/Cargo.toml as its crate manifest. The Dockerfile and docker/docker-compose.yml containers standardize the deployment environment across both languages.

What is the difference between the wiki/ and REALIGNMENT/ directories?

The wiki/ directory contains user-facing documentation, troubleshooting guides, and design notes intended for general consumption. The REALIGNMENT/ directory stores project-level architectural decisions, roadmaps, and internal design documents that guide the long-term evolution of the codebase.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}