How Does Caveman 2 Measure and Verify Token Savings?

Caveman 2 estimates token savings by parsing Claude session transcripts, attributing output tokens to specific modes using timestamped logs, and applying a benchmarked compression ratio of 0.65 to calculate what would have been generated without the tool.

The caveman CLI provides transparent analytics for Claude Code usage by measuring actual output tokens against a verified baseline. According to the JuliusBrussee/caveman source code, the tool implements a multi-stage pipeline that combines log parsing, mode detection, and historical benchmarking to produce accurate savings estimates.

Parsing Session Transcripts

The measurement process begins in src/hooks/caveman-stats.js with the parseSession function. This utility reads the *.jsonl transcript produced by Claude Code and extracts two critical metrics from each message's usage object: output_tokens and cache_read_input_tokens.

The function walks every line of the JSONL file, maintains a running total, and returns a timestamped list of per-message output tokens along with the aggregated counts. This raw data serves as the foundation for all subsequent savings calculations.

Detecting and Attributing Modes

Once the session data is parsed, Caveman 2 determines which compression ratio applies to each message.

Reading the Active Flag

The readFlag function in src/hooks/caveman-config.js inspects the .caveman-active flag file to identify the current mode (such as full). This file acts as the source of truth for which Caveman configuration was active during the session.

Mode Attribution Logic

The attributeByMode function joins the per-message timestamps from the transcript with the mode transition log stored in .caveman-mode-log.jsonl. This allows the system to handle mid-session mode switches accurately.

If the transition log is missing, the system falls back to a "flag-mtime" heuristic using the file modification time. In the absence of any historical data, it assumes the entire session used the current flag state. This attribution step ensures that savings calculations apply the correct compression ratio to each segment of the conversation.

Calculating Estimated Savings

Applying the Compression Benchmark

Caveman 2 stores benchmarked compression ratios in the COMPRESSION constant object:

const COMPRESSION = { full: 0.65 };

This value indicates that full mode reduces output tokens by approximately 65% compared to standard Claude Code usage. This ratio derives from empirical benchmarks recorded in the benchmarks/results/ directory.

The Savings Formula

The deriveSavings function implements the core calculation:

const saved = Math.round(tokens / (1 - ratio)) - tokens;

Where ratio is the compression factor (0.65 for full mode). This formula estimates the hypothetical token count that would have been required without Caveman, then subtracts the actual observed tokens to determine the savings.

USD Cost Conversion

The priceForModel function converts saved tokens into estimated dollar values using the MODEL_OUTPUT_PRICE_PER_M table, which contains per-model pricing data. This allows the CLI to display both token counts and monetary savings in its reports.

Verification Through Testing

The accuracy of these measurements relies on a comprehensive test suite located in tests/test_caveman_stats.js. This suite runs the actual caveman-stats script against synthetic sessions and asserts on exact output strings to verify:

• Correct token summation from JSONL transcripts
• Proper handling of the full compression ratio (0.65)
• Fallback behavior when benchmark data is missing
• Per-mode attribution accuracy, including mid-session mode changes
• USD conversion calculations using the price table
• Compression-pair detection algorithms
• Lifetime history recording and aggregation

By testing against synthetic data with known outcomes, the suite validates that the deriveSavings calculations and attributeByMode logic produce mathematically correct results across edge cases.

Reporting and Persistence

Human-Readable Output

The formatStats function generates the detailed multi-line report shown when running caveman-stats, while formatShare produces a concise one-liner optimized for social sharing. The --share flag explicitly invokes formatShare to output a single-line summary with emoji and estimated dollar savings.

Lifetime History Aggregation

Caveman 2 persists session snapshots to .caveman-history.jsonl. The aggregateHistory function later processes this file to support the --all and --since flags, collapsing multiple sessions into cumulative totals while avoiding duplicate entries.

Practical Usage Examples

View Session Statistics

caveman-stats

This command parses the most recent .jsonl transcript, applies mode attribution, and prints a detailed breakdown including output tokens, cache-read tokens, estimated savings, and USD value.

Generate a Shareable Summary

caveman-stats --share

Output:

🪨 Saved 650 output tokens (~$0.0098) across 1 turns this session — caveman.sh

View Lifetime Totals

caveman-stats --all

This aggregates all entries in .caveman-history.jsonl to display cumulative statistics across multiple sessions.

Programmatic Access

const { parseSession, attributeByMode, deriveSavings } = require('./src/hooks/caveman-stats.js');

const sess = parseSession('/path/to/session.jsonl');
const mode = require('./src/hooks/caveman-config.js').readFlag('/home/user/.claude/.caveman-active');
const modeLog = require('./src/hooks/caveman-stats.js').readModeLog('/home/user/.claude/.caveman-mode-log.jsonl');

const attribution = attributeByMode({
  messages: sess.messages,
  modeLog,
  mode,
  flagMtimeMs: null,
  outputTokens: sess.outputTokens,
});

const { estSavedTokens, estSavedUsd } = deriveSavings({ 
  byMode: attribution.byMode, 
  model: sess.model 
});

console.log(`Estimated saved tokens: ${estSavedTokens}`);
console.log(`Estimated USD saved:   ${estSavedUsd}`);

Summary

• Token parsing occurs via parseSession in src/hooks/caveman-stats.js, which extracts output_tokens and cache_read_input_tokens from Claude Code JSONL transcripts.
• Mode attribution uses attributeByMode to map timestamps to active modes, handling mid-session switches through .caveman-mode-log.jsonl.
• Savings calculation applies the formula Math.round(tokens / (1 - 0.65)) - tokens when full mode is active, based on the benchmarked 0.65 compression ratio.
• Verification relies on tests/test_caveman_stats.js, which validates summation logic, mode attribution, and USD conversion against synthetic data.
• Reporting functions formatStats and formatShare present results in both detailed and condensed formats, while aggregateHistory enables lifetime analytics.

Frequently Asked Questions

How does Caveman 2 handle mode switching mid-session?

Caveman 2 tracks mode transitions through the .caveman-mode-log.jsonl file. The attributeByMode function joins these timestamps with message timestamps from the transcript to apply the correct compression ratio to each segment of the conversation. If the log is missing, it falls back to the flag file's modification time or assumes the current mode for the entire session.

Where does the 0.65 compression ratio come from?

The 0.65 value for full mode (stored in the COMPRESSION constant) derives from empirical benchmarks stored in benchmarks/results/*.json. This ratio represents the measured reduction in output tokens when using Caveman compared to standard Claude Code usage without context optimization.

How accurate are the USD cost estimates?

The USD estimates use the MODEL_OUTPUT_PRICE_PER_M table in src/hooks/caveman-stats.js to calculate costs based on current per-model pricing. These values update with the tool's pricing data and reflect actual API rates, though they represent estimates unless matched to your specific billing tier.

Can I verify the token savings calculations independently?

Yes. The full calculation logic is exposed through the Node.js API in src/hooks/caveman-stats.js. You can import parseSession, attributeByMode, and deriveSavings to run the same calculations against your own transcript files. The test suite in tests/test_caveman_stats.js also serves as a reference for expected behavior with known input values.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}