How Benchmarks Are Measured in the `benchmarks/` and `evals/` Directories

The Caveman repository employs two complementary harnesses—benchmarks/ for direct token-count comparison of a single skill and evals/ for deterministic, multi-skill compression analysis—to quantify token savings using median-based statistical methods.

The JuliusBrussee/caveman project provides a systematic way to measure how system prompts reduce LLM output token counts. Understanding how benchmarks are measured in the benchmarks/ and evals/ directories reveals the rigorous methodology behind the repository's token-saving claims.

The Two Measurement Harnesses

The repository splits measurement into two distinct approaches: one for quick validation of a single skill, and another for comprehensive evaluation across all available skills.

benchmarks/ - Direct Token Comparison

The benchmarks/ directory contains a harness that measures direct token counts for a single skill against a normal Claude system prompt. The core implementation lives in [benchmarks/run.py](https://github.com/JuliusBrussee/caveman/blob/main/benchmarks/run.py), where the run_benchmarks function (lines 87‑114) drives the measurement loop.

The process follows these steps:

1. Load prompts from [benchmarks/prompts.json](https://github.com/JuliusBrussee/caveman/blob/main/benchmarks/prompts.json).
2. For each prompt, call the Anthropic API twice—once with the vanilla system prompt ("You are a helpful assistant.") and once with the Caveman system prompt (the content of [skills/caveman/SKILL.md](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman/SKILL.md)).
3. Repeat each call for a configurable number of trials (default 3).
4. Compute the median output-token count per mode using the compute_stats function (lines 117‑148).
5. Derive the percentage savings using the formula: 1 - (caveman_median / normal_median).
6. Persist raw results, a summary table, and JSON artifacts under benchmarks/results/.
7. Optionally rewrite the markdown table in [README.md](https://github.com/JuliusBrussee/caveman/blob/main/README.md) between the <!-- BENCHMARK-TABLE-START --> markers via the update_readme function (lines 214‑229).

evals/ - Real-World Compression Across Skills

The evals/ directory provides a deterministic snapshot-based evaluation that measures real-world token compression across many skills. This harness uses a three-arm design to establish proper baselines.

The workflow is implemented across two files:

• [evals/llm_run.py](https://github.com/JuliusBrussee/caveman/blob/main/evals/llm_run.py) (lines 43‑99): Generates the snapshot
• [evals/measure.py](https://github.com/JuliusBrussee/caveman/blob/main/evals/measure.py) (lines 29‑101): Analyzes the results

The measurement process works as follows:

1. Read the prompt list from [evals/prompts/en.txt](https://github.com/JuliusBrussee/caveman/blob/main/evals/prompts/en.txt) and discover every skill directory under skills/.
2. For each prompt, run the Claude CLI three times:
   • Baseline: No system prompt
   • Terse: "Answer concisely."
   • Terse+Skill: The terse prefix plus the skill’s markdown content
3. Write all raw LLM outputs to [evals/snapshots/results.json](https://github.com/JuliusBrussee/caveman/blob/main/evals/snapshots/results.json), a version-controlled snapshot.
4. Load the snapshot in measure.py and tokenize each output using the OpenAI tiktoken o200k_base encoder (an approximate Claude tokenizer).
5. Compute per-skill savings relative to the terse control using: savings = 1 - (skill_tokens / terse_tokens).
6. Print a markdown table showing median, mean, min, max, and standard deviation of the savings, plus raw token totals.

According to the [evals/README.md](https://github.com/JuliusBrussee/caveman/blob/main/evals/README.md), the terse-control arm serves as the proper baseline because it isolates the skill's contribution beyond mere conciseness instructions.

Running the Benchmarks

Execute the single-skill benchmark with configurable trials and optional README updates:

python benchmarks/run.py \
  --trials 3 \
  --model claude-sonnet-4-20250514 \
  --update-readme

Validate the benchmark logic without consuming API credits:

python benchmarks/run.py --dry-run --trials 2

Run the full multi-skill evaluation pipeline:

# Generate the snapshot (requires Claude CLI)

uv run python evals/llm_run.py

# Analyze token compression

uv run --with tiktoken python evals/measure.py

Key Implementation Details

The benchmarks/run.py script uses median rather than mean calculations to mitigate outliers from API variance. The compute_stats function aggregates trial data and computes the final savings percentage before the update_readme function injects results back into the project documentation.

In the evals/ harness, the snapshot architecture separates data generation from analysis. This allows the [evals/snapshots/results.json](https://github.com/JuliusBrussee/caveman/blob/main/evals/snapshots/results.json) to serve as a version-controlled source of truth, enabling reproducible measurements across different versions of the tokenizer or analysis logic.

Summary

• Two complementary approaches: benchmarks/ for single-skill validation, evals/ for comprehensive multi-skill analysis.
• Median-based statistics: Both harnesses use median calculations to ensure robust measurements against API variance.
• Three-arm evaluation: The evals/ harness compares skills against both baseline and terse controls to isolate true compression value.
• Deterministic snapshots: The evaluation harness version-controls raw LLM outputs in evals/snapshots/results.json for reproducible analysis.
• Automated reporting: The benchmark harness can automatically update README tables with fresh results via the --update-readme flag.

Frequently Asked Questions

What is the difference between the benchmarks/ and evals/ directories?

The benchmarks/ directory provides a quick, single-signal measurement comparing one specific skill (Caveman) against a vanilla system prompt using live API calls. The evals/ directory implements a comprehensive, deterministic evaluation that tests all skills against both baseline and terse controls, storing results in a version-controlled snapshot for reproducible analysis.

How does the evaluation calculate token savings percentages?

Both harnesses use a percentage savings formula based on median token counts. The benchmarks/ harness calculates 1 - (caveman_median / normal_median), while the evals/ harness calculates 1 - (skill_tokens / terse_tokens) relative to the terse control arm. This approach measures the proportional reduction in output tokens compared to the respective baseline.

Why does the evals harness use a terse control instead of a baseline?

According to the [evals/README.md](https://github.com/JuliusBrussee/caveman/blob/main/evals/README.md), the terse control ("Answer concisely.") serves as the proper baseline because it isolates the specific contribution of the skill markdown beyond generic conciseness instructions. Comparing against the terse control reveals the additional token savings provided by the structured skill content itself.

Can I run the benchmarks without making actual API calls?

Yes. The benchmarks/run.py script supports a --dry-run flag that simulates the API calls and displays the planned execution flow without contacting the Anthropic service. However, the evals/llm_run.py harness requires the Claude CLI to generate the deterministic snapshot, as it relies on actual LLM outputs for tokenization analysis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}