BiomeJS Performance Benchmarks: A Technical Deep Dive into the Criterion-Based Testing Architecture

BiomeJS performance benchmarks use Criterion to measure parser, formatter, and analyzer throughput on real-world JavaScript libraries, caching inputs under target/ and reporting bytes-per-second metrics.

The biomejs/biome repository ships with a comprehensive suite of micro-benchmarks designed to validate the performance of its JavaScript and TypeScript tooling. These benchmarks exercise the core stages of the pipeline—parsing, formatting, and analysis—using real-world library code downloaded on-the-fly. Understanding the BiomeJS performance benchmarks architecture helps contributors optimize the toolchain and gives users confidence in the engine's throughput characteristics.

Benchmark Architecture and Pipeline Stages

The Biome codebase organizes benchmarks as Criterion test suites spread across multiple crates. Each major stage of the JavaScript processing pipeline has its own dedicated benchmark driver:

Parsing Benchmarks

The JavaScript and TypeScript parser benchmarks live in crates/biome_js_parser/benches/js_parser.rs. This file defines two measurement modes:

• Uncached parsing: Measures biome_js_parser::parse with cold inputs
• Cached parsing: Measures parse_js_with_cache for repeated analysis scenarios

Each input file becomes a separate BenchmarkId, with throughput set to the byte size of the source to enable bytes/s reporting.

Formatting Benchmarks

Located at crates/biome_js_formatter/benches/js_formatter.rs, the formatter benchmarks parse the input once, then repeatedly invoke biome_js_formatter::format_node on the resulting AST. This isolates formatting latency from parsing overhead and measures the actual layout engine performance.

Analysis Benchmarks

The static analysis pipeline is benchmarked in crates/biome_js_analyze/benches/js_analyzer.rs. Following the same pattern as the parser and formatter, this driver exercises the biome_js_analyze::analyze function against real-world codebases to detect performance regressions in lint rules and semantic analysis.

The BenchCase Helper and Input Management

All benchmarks rely on the BenchCase type defined in crates/biome_test_utils/src/bench_case.rs. This helper handles the logistics of benchmark input:

1. URL Resolution: Each benchmark suite points to a text file (e.g., libs-js.txt, libs-ts.txt) containing URLs of popular npm packages hosted on GitHub
2. Deterministic Caching: For each URL, the helper resolves a deterministic filename, downloads the content via ureq, and stores it under target/<hash>.js
3. Isolation: This caching strategy guarantees repeatable runs while isolating network latency from the measured numbers

The BenchCase exposes methods like code() and filename() that benchmarks use to access the source text and metadata.

Memory Allocation and Statistical Rigor

To eliminate allocator noise on large inputs, the benchmark binaries configure a fast global allocator using #[cfg] guards:

• Windows: mimalloc
• Linux/macOS: jemalloc

This ensures that memory management overhead does not skew the results when processing large JavaScript bundles. Criterion groups are created per stage (e.g., "js_parser"), with each input file receiving its own BenchmarkId and throughput measurement based on file size.

Running Benchmarks Locally

The benchmarks are ordinary Rust binaries generated by Cargo. After cloning the repository, build all benchmarks without running them:

cargo bench --all --no-run

To execute a specific benchmark suite:

# Parse JavaScript and TypeScript sources

cargo bench --bench js_parser

# Format the parsed ASTs

cargo bench --bench js_formatter

# Run the analyzer pipeline

cargo bench --bench js_analyzer

For CI integration or more stable local numbers, tune Criterion through environment variables:

# Increase sample size for statistical stability

CRITERION_SAMPLE_SIZE=30 cargo bench --bench js_parser

The output displays time, mean, standard deviation, and throughput (bytes per second). Criterion also emits JSON reports under target/criterion for automated tracking.

Adding Custom Benchmarks

To measure a new language feature or specific code pattern, extend the existing infrastructure. First, add a URL to the appropriate suite file (e.g., crates/biome_js_parser/benches/libs-js.txt):

https://raw.githubusercontent.com/some/package/v1.2.3/src/optional_chaining.js

Then create a minimal benchmark harness:

use biome_js_parser::{parse, JsParserOptions};
use biome_languages::JsFileSource;
use biome_test_utils::BenchCase;
use criterion::{Criterion, BenchmarkId, Throughput, black_box, criterion_group, criterion_main};

fn bench_custom_parser(c: &mut Criterion) {
    let case = BenchCase::try_from(
        "https://raw.githubusercontent.com/some/package/v1.2.3/src/optional_chaining.js"
    ).expect("download case");
    let code = case.code();
    let src = JsFileSource::default();

    let mut group = c.benchmark_group("custom_js_parser");
    group.throughput(Throughput::Bytes(code.len() as u64));
    group.bench_with_input(
        BenchmarkId::new("optional_chaining", case.filename()),
        &code,
        |b, _| b.iter(|| black_box(parse(code, src, JsParserOptions::default()))),
    );
    group.finish();
}

criterion_group!(custom_parser, bench_custom_parser);
criterion_main!(custom_parser);

Run the new benchmark with:

cargo bench --bench custom_parser

Summary

• BiomeJS performance benchmarks are organized as Criterion test suites across biome_js_parser, biome_js_formatter, and biome_js_analyze crates
• The BenchCase helper in crates/biome_test_utils/src/bench_case.rs manages real-world library downloads and caching under target/
• Benchmarks use platform-specific fast allocators (mimalloc on Windows, jemalloc on Unix) to eliminate memory management noise
• Throughput is measured in bytes per second, providing realistic metrics for JavaScript processing speed
• Execute specific suites with cargo bench --bench <name> and tune stability via CRITERION_SAMPLE_SIZE

Frequently Asked Questions

How does BiomeJS ensure benchmark results are reproducible?

The BenchCase type downloads source files from URLs listed in suite files like libs-js.txt, then caches them under target/<hash>.js using deterministic filenames. This caching strategy ensures that network latency does not affect measurements and that subsequent runs process identical inputs.

What allocator does Biome use for benchmarking?

The benchmark binaries configure a fast global allocator using conditional compilation. On Windows, they use mimalloc; on Linux and macOS, they use jemalloc. This prevents standard allocator overhead from skewing performance measurements on large JavaScript files.

Can I benchmark Biome against my own JavaScript code?

Yes. You can add your own URLs to the library list files (e.g., libs-js.txt) or create a custom benchmark harness that uses BenchCase::try_from() to fetch your code. The BenchCase API provides code() and filename() methods to integrate your sources into Criterion benchmarks.

Where does Biome store benchmark results?

Criterion outputs results to the console as formatted tables showing time, mean, standard deviation, and throughput. Additionally, JSON reports are emitted under target/criterion for programmatic analysis and CI integration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}