How to Configure Token Counting and Usage Tracking in AgentScope

AgentScope provides a pluggable architecture for token counting and usage tracking through the TokenCounterBase interface, TruncatedFormatterBase integration, and automatic OpenTelemetry tracing of ChatUsage metrics.

Configuring token counting and usage tracking in AgentScope allows you to monitor LLM consumption, enforce context window limits, and export usage metrics to observability platforms. The agentscope-ai/agentscope repository implements this through a decoupled architecture where token counters are injected into message formatters, while usage tracking flows through the tracing subsystem.

Understanding AgentScope's Token Counting Architecture

AgentScope separates token counting from model logic through an abstract base class and concrete provider-specific implementations.

The TokenCounterBase Abstract Interface

All token counters inherit from TokenCounterBase defined in src/agentscope/token/_token_base.py. This interface requires a single async method:

async def count(self, messages: list[dict], **kwargs) -> int

The count method receives a list of message dictionaries (standard OpenAI-style format) and returns the total token count as an integer. This abstraction allows AgentScope to support any LLM provider or custom counting logic without modifying formatter code.

Built-in Counter Implementations

AgentScope ships with several concrete implementations in src/agentscope/token/:

• OpenAITokenCounter (_openai_token_counter.py): Uses tiktoken to compute exact token counts for OpenAI models, including vision model image token calculations via _get_base_and_tile_tokens.
• CharTokenCounter (_char_token_counter.py): Provides lightweight character-based counting for debugging or approximate limits without external dependencies.
• AnthropicTokenCounter, GeminiTokenCounter, HuggingFaceTokenCounter: Provider-specific implementations for non-OpenAI APIs.

Configuring Token Counters in Formatters

Token counters are injected into formatters that inherit from TruncatedFormatterBase in src/agentscope/formatter/_truncated_formatter_base.py.

Using OpenAITokenCounter with TruncatedFormatterBase

The formatter uses the counter to enforce max_tokens limits through an automatic truncation loop. When format() is called:

1. The formatter builds the provider-specific payload via _format()
2. It calls _count() which invokes await self.token_counter.count(formatted_msgs) (lines 73-78)
3. If the count exceeds max_tokens, it truncates oldest non-system messages via _truncate() (lines 81-84)

from agentscope.token import OpenAITokenCounter
from agentscope.formatter import OpenAIFormatter

# Initialize counter for specific model

counter = OpenAITokenCounter(model_name="gpt-4o-mini")

# Create formatter with 2048 token limit

formatter = OpenAIFormatter(token_counter=counter, max_tokens=2048)

# Use in your agent - truncation happens automatically

formatted_messages = await formatter.format(conversation_history)

Lightweight Character-Based Counting

For development or scenarios where exact token counts are unnecessary, use CharTokenCounter to avoid tiktoken dependencies:

from agentscope.token import CharTokenCounter
from agentscope.formatter import OpenAIFormatter

char_counter = CharTokenCounter()
formatter = OpenAIFormatter(token_counter=char_counter, max_tokens=4000)

Enabling Usage Tracking and Tracing

While token counting prevents context overflow, usage tracking records actual consumption after LLM calls complete.

The ChatUsage Data Model

Every model response in AgentScope includes a ChatUsage object defined in src/agentscope/model/_model_usage.py. This dataclass contains:

• input_tokens: Tokens sent to the model
• output_tokens: Tokens generated by the model
• total_tokens: Sum of input and output (optional)
• model: Model identifier
• metadata: Provider-specific extra fields

Model implementations in src/agentscope/model/ (e.g., _openai_model.py) extract usage data from provider responses and populate ChatUsage automatically:

# Response includes usage metadata

response = await model(formatted_messages)
print(f"Input: {response.usage.input_tokens}")
print(f"Output: {response.usage.output_tokens}")

OpenTelemetry Integration

AgentScope exports usage metrics as OpenTelemetry GenAI semantic conventions. The tracing system in src/agentscope/tracing/_extractor.py (lines 374-380) maps ChatUsage fields to standard attributes:

• GEN_AI_USAGE_INPUT_TOKENS (gen_ai.usage.input_tokens)
• GEN_AI_USAGE_OUTPUT_TOKENS (gen_ai.usage.output_tokens)

These attributes are defined in src/agentscope/tracing/_attributes.py (lines 65-70).

Enable tracing by wrapping model calls with the trace_chat decorator:

from agentscope.tracing import trace_chat
from agentscope.model import OpenAIChatModel

model = OpenAIChatModel(model_name="gpt-4o-mini")

@trace_chat
async def generate_summary(prompt: str):
    messages = [{"role": "user", "content": prompt}]
    return await model(messages)

# Usage metrics automatically exported to configured OTLP endpoint

response = await generate_summary("Explain async programming")

Complete Working Examples

Example 1: Production Setup with Token Limits and Tracing

import asyncio
from agentscope.token import OpenAITokenCounter
from agentscope.formatter import OpenAIFormatter
from agentscope.model import OpenAIChatModel
from agentscope.tracing import trace_chat, setup_tracing

async def main():
    # Setup token counter and formatter with 4k limit

    counter = OpenAITokenCounter(model_name="gpt-4o")
    formatter = OpenAIFormatter(token_counter=counter, max_tokens=4096)
    
    # Initialize model

    model = OpenAIChatModel(model_name="gpt-4o")
    
    @trace_chat
    async def chat(messages):
        formatted = await formatter.format(messages)
        return await model(formatted)
    
    # Conversation that might exceed limit

    long_history = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a very long story."},
        # ... many more messages

    ]
    
    response = await chat(long_history)
    print(f"Tokens: {response.usage.input_tokens} in, {response.usage.output_tokens} out")

if __name__ == "__main__":
    asyncio.run(main())

Example 2: Custom Token Counter for Specialized Models

from agentscope.token import TokenCounterBase
from agentscope.formatter import OpenAIFormatter

class WordTokenCounter(TokenCounterBase):
    """Simple counter that splits on whitespace - useful for local models."""
    
    async def count(self, messages, **kwargs):
        total = 0
        for msg in messages:
            content = msg.get("content", "")
            # Count words as proxy for tokens

            total += len(content.split())
        return total

# Use custom counter

word_counter = WordTokenCounter()
formatter = OpenAIFormatter(token_counter=word_counter, max_tokens=500)

Summary

• Token counting in AgentScope is implemented via the TokenCounterBase abstract class in src/agentscope/token/_token_base.py, with concrete implementations like OpenAITokenCounter providing provider-specific logic.
• Formatter integration occurs through TruncatedFormatterBase in src/agentscope/formatter/_truncated_formatter_base.py, which automatically truncates messages when token_counter.count() exceeds max_tokens.
• Usage tracking relies on the ChatUsage dataclass in src/agentscope/model/_model_usage.py, populated by model implementations and extracted by the tracing system in src/agentscope/tracing/_extractor.py.
• OpenTelemetry export maps usage to standard GenAI attributes (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens) defined in src/agentscope/tracing/_attributes.py, enabling observability across all supported LLM providers.

Frequently Asked Questions

How do I switch from OpenAI's tokenizer to a character-based counter?

Replace OpenAITokenCounter with CharTokenCounter from src/agentscope/token/_char_token_counter.py. This implementation counts characters instead of tokens, providing a lightweight alternative that doesn't require the tiktoken library. Simply instantiate CharTokenCounter() and pass it to your formatter's token_counter parameter.

Can I use token counting without enabling tracing?

Yes. Token counting via TokenCounterBase implementations and TruncatedFormatterBase operates independently of the tracing system. The formatter's truncation logic calls token_counter.count() during message formatting to enforce max_tokens limits, regardless of whether you wrap your model calls with @trace_chat or other tracing decorators.

Where does AgentScope store the actual token usage after an LLM call?

Token usage is stored in the usage attribute of the ChatResponse object returned by model implementations. This attribute contains a ChatUsage instance (defined in src/agentscope/model/_model_usage.py) with input_tokens and output_tokens fields. Model classes in src/agentscope/model/ populate this data from provider-specific response fields like OpenAI's response.usage or Anthropic's usage metadata.

How do I create a custom token counter for a model not supported by AgentScope?

Implement the TokenCounterBase abstract class from src/agentscope/token/_token_base.py by creating an async count(self, messages, **kwargs) method that returns an integer token count. Your implementation can use any tokenization logic appropriate for your model (e.g., HuggingFace tokenizers, character counts, or word counts). Once implemented, pass an instance of your custom counter to any formatter inheriting from TruncatedFormatterBase via the token_counter parameter.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}