Token Usage Calculation Methodology in ConversationAnalyzer: A Technical Deep Dive

The ConversationAnalyzer class implements a multi-tiered token usage calculation methodology that prioritizes cached results, then aggregates input_tokens, output_tokens, cache_creation_input_tokens, and cache_read_input_tokens from each message's usage object, falling back to a character-based estimate for legacy conversation files.

The ConversationAnalyzer in the davila7/claude-code-templates repository provides precise token accounting for Claude Code interactions. Located at cli-tool/src/analytics/core/ConversationAnalyzer.js, this analyzer processes conversation JSONL files to determine exactly how many tokens each exchange consumed, enabling accurate usage statistics for dashboards and CLI reporting tools.

The Four-Step Calculation Process

The token usage calculation methodology follows a hierarchical approach that optimizes for performance while maintaining accuracy across different conversation formats.

Step 1: Cache Retrieval via getCachedTokenUsage()

Before parsing conversation files, the analyzer checks for cached calculations to avoid redundant processing of large logs. On line 111, the code calls getCachedTokenUsage(), which references a cache helper implementation on line 321. If a valid cached entry exists for the conversation file, the method returns the stored token counts immediately, bypassing expensive file I/O and computation.

Step 2: Real-Time Aggregation in calculateRealTokenUsage()

When no cache entry exists, the analyzer walks through every message in the conversation and sums the token counters that Claude returns in each message's usage object. This logic resides in calculateRealTokenUsage() (lines 380-401) within cli-tool/src/analytics/core/ConversationAnalyzer.js.

The method initializes counters for each token type:

let totalInputTokens = 0;
let totalOutputTokens = 0;
let totalCacheCreationTokens = 0;
let totalCacheReadTokens = 0;

for (const message of parsedMessages) {
  totalInputTokens += message.usage.input_tokens || 0;
  totalOutputTokens += message.usage.output_tokens || 0;
  totalCacheCreationTokens += message.usage.cache_creation_input_tokens || 0;
  totalCacheReadTokens += message.usage.cache_read_input_tokens || 0;
}

return {
  total: totalInputTokens + totalOutputTokens,
  inputTokens: totalInputTokens,
  outputTokens: totalOutputTokens,
  cacheCreationTokens: totalCacheCreationTokens,
  cacheReadTokens: totalCacheReadTokens,
};

Step 3: Character-Based Fallback with estimateTokens()

If the conversation file predates the usage telemetry format and contains no usage payload, the analyzer falls back to a rough estimation assuming approximately 4 characters per token. The estimateTokens() method (lines 828-833) implements this heuristic, invoked on line 128 when tokenUsage.total equals zero.

Step 4: Object Integration

Finally, the methodology attaches the calculated data to the conversation record during the processing loop (lines 99-104). Every conversation entry receives both the granular breakdown and a safe total:

tokens: tokenUsage.total > 0
         ? tokenUsage.total
         : this.estimateTokens(await this.getFileContent(filePath)),
tokenUsage: tokenUsage,

Handling Legacy Conversation Formats

The token usage calculation methodology accounts for historical data through its fallback mechanism. When calculateRealTokenUsage() returns zeros for all counters, the analyzer switches to estimateTokens(), which divides the total character count by 4 to approximate token consumption. While less precise than the telemetry-based approach, this ensures that older conversation logs still contribute to aggregate statistics rather than being discarded as null values.

Complete Working Example

The following implementation demonstrates how to instantiate the analyzer and retrieve token statistics using the supporting StateCalculator and ProcessDetector classes:

const ConversationAnalyzer = require('./cli-tool/src/analytics/core/ConversationAnalyzer');
const StateCalculator = require('./cli-tool/src/analytics/StateCalculator');
const ProcessDetector = require('./cli-tool/src/analytics/ProcessDetector');

(async () => {
  const analyzer = new ConversationAnalyzer('/home/user/.claude');
  const stateCalc = new StateCalculator();
  const procDet = new ProcessDetector();

  const data = await analyzer.loadInitialData(stateCalc, procDet);
  console.log('Total tokens across all conversations:', data.summary.totalTokens);
  console.log('First conversation token breakdown:', data.conversations[0].tokenUsage);
})();

This script loads all *.jsonl files from the specified Claude directory, computes (or retrieves from cache) the token usage for each conversation, and exposes the detailed breakdown through the tokenUsage property on each conversation object.

Summary

• The ConversationAnalyzer in davila7/claude-code-templates implements a cache-first token usage calculation methodology that minimizes redundant processing of large conversation files.
• Real-time calculation aggregates four distinct counters—input_tokens, output_tokens, cache_creation_input_tokens, and cache_read_input_tokens—within the calculateRealTokenUsage() method.
• Legacy support falls back to character-based estimation (4 characters per token) via estimateTokens() when usage telemetry is unavailable.
• Integration attaches both granular token counts and safe estimates to every conversation record, ensuring comprehensive analytics regardless of log format.

Frequently Asked Questions

How does ConversationAnalyzer handle conversations without usage telemetry?

When message objects lack a usage property, the analyzer invokes estimateTokens() (lines 828-833), which applies a heuristic of approximately 4 characters per token to the raw file content. This fallback activates on line 128 when tokenUsage.total is zero, ensuring that older conversation logs still contribute to aggregate statistics rather than returning null values.

What specific token counters does the methodology aggregate?

The calculateRealTokenUsage() method sums four specific fields from each message's usage object: input_tokens (prompt tokens sent to the model), output_tokens (completion tokens generated by the model), cache_creation_input_tokens (tokens that triggered cache creation), and cache_read_input_tokens (tokens retrieved from cache). These counters are summed independently and returned as a structured object containing both individual counts and a grand total.

How is token usage cached between analysis runs?

The analyzer implements a DataCache helper (referenced on line 321) that stores computed token results via getCachedTokenUsage(). When loadInitialData() processes a conversation file, it first checks line 111 for an existing cache entry; if found, it returns the cached value immediately, avoiding the need to re-parse large JSONL files and recalculate sums, significantly improving performance on subsequent runs.

What is the performance impact of parsing large conversation histories?

The token usage calculation methodology mitigates performance costs through two mechanisms: caching eliminates redundant parsing of unchanged files, and the streaming JSONL parser processes messages sequentially rather than loading entire conversations into memory. For files without cache entries, the algorithm runs in O(n) time relative to message count, with the calculateRealTokenUsage() loop (lines 380-401) executing simple arithmetic operations that impose minimal overhead even on extensive chat histories.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}