How CodeCompressor Performs AST-Aware Compression for Different Languages

The CodeCompressor in the Headroom repository uses tree-sitter to parse source code into an abstract syntax tree (AST), then applies language-specific node configurations to identify structural elements and prune low-value symbols while preserving syntactic validity across Python, Rust, JavaScript, and other supported languages.

The CodeCompressor class in headroom/transforms/code_compressor.py implements sophisticated AST-aware compression that goes beyond simple text-based minification. According to the Headroom source code, this system leverages the tree-sitter parsing library to understand code structure at the semantic level, enabling intelligent reduction that maintains valid syntax for each target language.

How Tree-Sitter Powers AST-Aware Compression

The foundation of AST-aware compression rests on robust parsing infrastructure that handles multiple languages through a unified interface.

Checking Parser Availability

Before attempting any AST operations, the compressor verifies that the tree-sitter dependency is present. The _check_tree_sitter_available() function performs this validation, while is_tree_sitter_available() provides a cached boolean check for subsequent operations. This ensures graceful degradation when the optional dependency is missing.

Language Detection and Parser Selection

When processing raw source code, the detect_language() function analyzes content using regex hints—such as def and class for Python or #include for C-style languages—to determine the programming language and return a confidence score. Once identified, the _get_parser() function retrieves a cached tree_sitter.Language object specific to the detected language, enabling subsequent AST generation.

Language-Specific AST Configuration

The compressor maintains distinct node type mappings for each supported language, allowing it to recognize structural elements regardless of syntactic differences.

The configuration block beginning at line 218 defines per-language sets such as function_nodes, class_nodes, decorator_node, and detection hints. For Python specifically (lines 232-239), the configuration specifies:

• function_nodes=frozenset({"function_definition"})
• class_nodes=frozenset({"class_definition"})
• decorator_node="decorated_definition"

These mappings inform the _extract_structure() method, which populates a CodeStructure dataclass defined at lines 327-337. This dataclass organizes imports, top-level definitions, function signatures, type declarations, and comments into separate categorized lists for further processing.

The Compression Pipeline

The AST-aware compression follows a multi-stage pipeline that analyzes symbol importance before restructuring the code.

Extracting Code Structure

The _extract_structure() method traverses the tree-sitter AST, identifying nodes based on the language-specific configuration. It separates structural elements from comments and whitespace, creating a semantic map of the source file that preserves relationships between imports, classes, and functions.

Analyzing Symbol Importance

The _analyze_symbol_importance() function evaluates each symbol—functions, classes, and variables—based on usage frequency, call sites, and import dependencies. This analysis generates a symbol_scores mapping (referenced at line 428) that quantifies the value of each code element relative to the overall comprehension of the file.

Allocating Compression Budget

Using the importance scores, _allocate_body_budget() distributes a global token budget across the file's definitions. High-priority symbols receive larger allocations, while low-value or unused functions are marked for reduction or removal. This budget-driven approach ensures the compressed output retains the most semantically significant code within size constraints.

AST-Guided Pruning

For individual definitions, the compressor applies language-aware pruning strategies. The _compress_function_ast() method truncates or elides function bodies based on their budget allocation, while _compress_class_ast() handles class definitions and their member hierarchies. Both methods preserve required syntactic elements—such as decorators, type signatures, and indentation—ensuring the output remains parseable.

The main entry point compress() orchestrates this pipeline, delegating to _compress_with_ast() at line 1106 when tree-sitter processing is available, ultimately returning a DiffCompressionResult containing the compressed source string.

Integration with the Content Router

The CodeCompressor integrates with Headroom's transformation system through headroom/transforms/content_router.py. When enable_code_aware=True is configured, the router automatically instantiates the CodeAwareCompressor for detected code payloads, falling back to text-based compressors like LogCompressor when the content is not recognized as source code. This routing mechanism ensures that AST-aware processing applies only to appropriate content types without manual intervention.

Practical Examples

The following examples demonstrate AST-aware compression for Python and Rust code:

from headroom.transforms import CodeAwareCompressor

# Example 1: Compress a Python snippet

compressor = CodeAwareCompressor()
code = """
import numpy as np
def heavy_compute(x):
    # long comment …

    total = 0
    for i in range(1000):
        total += x * i
    return total
"""
result = compressor.compress(code, language="python")
print(result.compressed)          # Keeps imports, drops the heavy body

print(result.compression_ratio)  # e.g., 0.48 (48% size reduction)

from headroom.transforms import CodeAwareCompressor

# Example 2: Compress a Rust module

compressor = CodeAwareCompressor()
rust_code = """
use std::collections::HashMap;
pub struct Data { pub id: i32, pub payload: String }
impl Data {
    pub fn new(id: i32, payload: String) -> Self { Self { id, payload } }
    pub fn compute(&self) -> i32 { (0..1000).map(|i| i * self.id).sum() }
}
"""
result = compressor.compress(rust_code, language="rust")
print(result.compressed)  # Stripped the compute loop while keeping the struct

Both examples utilize the _compress_with_ast() path, ensuring the output maintains valid syntax for the target language.

Summary

• Tree-sitter integration provides the parsing foundation that enables language-aware AST manipulation across multiple programming languages.
• Language-specific configurations map tree-sitter node types to semantic categories (functions, classes, imports) for Python, Rust, JavaScript, and other supported languages.
• Symbol importance analysis scores code elements based on usage patterns and dependencies to prioritize retention of critical logic.
• Budget allocation distributes token limits across definitions, allowing high-priority symbols to retain full implementations while pruning low-value code.
• Syntactic preservation ensures that compressed output remains parseable and valid, maintaining decorators, type signatures, and structural indentation.

Frequently Asked Questions

What is AST-aware compression?

AST-aware compression is a technique that parses source code into an abstract syntax tree before applying reductions. Unlike text-based minification that blindly removes whitespace or comments, AST-aware compression understands code structure, allowing it to remove entire functions or truncate implementation details while preserving the syntactic validity and semantic relationships of the remaining code.

Which languages does CodeCompressor support?

The CodeCompressor supports any language available in the tree-sitter ecosystem, with explicit configuration blocks for Python, JavaScript, TypeScript, Rust, C, C++, and Go. The language detection mechanism at line 522 uses regex hints to identify the source language, while the parser selection at line 76 retrieves the appropriate tree-sitter grammar for that specific language.

How does the compressor maintain syntactic validity?

The compressor maintains validity by operating directly on AST nodes rather than raw text. Methods like _compress_function_ast() and _compress_class_ast() preserve required syntactic elements—including decorators, function signatures, class declarations, and indentation—while only modifying or removing body content. The reassembly process reconstructs the source file from the modified AST, ensuring the output conforms to the language grammar.

What happens if tree-sitter is not installed?

If tree-sitter is unavailable, the is_tree_sitter_available() check at line 139 returns False, causing the compressor to bypass AST-based processing. The system falls back to alternative compression strategies or returns the original content uncompressed, ensuring the application continues to function without the optional dependency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}