Framework Test Patterns for Testing aisuite Applications: A Complete Guide

aisuite applications are tested through a provider-agnostic framework layer that validates Pydantic message models, maps ASR parameters, and verifies agent tracing through mocked Runner workflows.

The aisuite library (andrewyng/aisuite) unifies multiple LLM providers behind a common Python interface. Mastering the framework test patterns for testing aisuite applications allows you to validate chat completions, transcription workflows, and multi-agent orchestration without hitting live APIs.

Testing Unified Message Objects

The foundation of aisuite testing rests on the Message class and its related tool-call models defined in aisuite/framework/message.py (lines 26‑33). These Pydantic models represent the OpenAI‑style payload that travels between your application and provider clients.

Tests construct Message instances with tool_calls lists to simulate LLM responses. A typical pattern involves creating a ChatCompletionMessageToolCall containing a Function object, then passing this message into Runner.run_sync or a mock provider to assert that the resulting output contains expected content.

from aisuite.framework.message import Message, ChatCompletionMessageToolCall, Function
import aisuite as ai

def make_tool_call(name, args, call_id):
    return ChatCompletionMessageToolCall(
        id=call_id,
        type="function",
        function=Function(name=name, arguments=args),
    )

def test_tool_call_flow():
    # Mock LLM response that requests a tool

    response_msg = Message(
        role="assistant",
        tool_calls=[make_tool_call("search", '{"query":"AI"}', "c1")]
    )
    # Feed it into the runner (provider is mocked elsewhere)

    result = ai.Runner.run_sync(
        ai.Agent(name="test", model="openai:gpt-4o", tools=[]),
        "dummy",
        client=ai.Client(),
        initial_message=response_msg,
    )
    assert result.final_output is not None

Validating ASR Data Models

Automatic Speech Recognition (ASR) workflows rely on a separate set of Pydantic models in the framework layer. The test suite in tests/framework/test_asr_models.py (lines 15‑40) validates Word, Segment, Alternative, Channel, and TranscriptionResult classes.

Tests instantiate these models with minimal required fields and use pydantic.ValidationError to enforce schema compliance. Optional provider‑specific fields such as confidence and speaker are tested for correct round‑trip serialization.

Parameter Mapping with ParamValidator

The ParamValidator class in aisuite/framework/asr_params.py (lines 17‑38) maps common OpenAI‑style parameters (language, prompt, temperature) to provider‑specific equivalents. The corresponding tests in tests/framework/test_asr_params.py (lines 13‑70) verify this translation logic.

Tests call ParamValidator.validate_and_map(provider, params) with raw dictionaries and assert that returned dicts contain correctly mapped keys. For example, Google translates language to language_code. The test suite also exercises the three extra_param_mode behaviors—strict, warn, and permissive—to ensure invalid parameters are handled according to configuration.

from aisuite.framework.asr_params import ParamValidator

def test_google_language_mapping():
    validator = ParamValidator(extra_param_mode="strict")
    mapped = validator.validate_and_map("google", {"language": "en"})
    assert mapped == {"language_code": "en-US"}   # Google expands 2‑letter codes

Agent Integration and Tracing Flows

End‑to‑end agent testing occurs in tests/agents/test_agent_integration_flows.py (lines 16‑63). These tests orchestrate Agent → Sub‑Agent chains using ai.agent_tool(subagent, name=...) and verify that the JsonlTraceStore captures parent/child run relationships.

Tests mock the provider’s chat_completions_create method to return pre‑crafted Message objects containing tool calls. After executing Runner.run_sync, assertions verify that the trace store contains both runs with correct parent_run_id linkages.

import aisuite as ai
from aisuite.tracing import LocalTraceSink, JsonlTraceStore

def test_parent_child_trace(tmp_path):
    client = ai.Client()
    client.providers["openai"] = Mock()
    # ...set side_effect on provider.chat_completions_create as in the repo...

    sink = LocalTraceSink(tmp_path / "trace.jsonl")
    parent = ai.Runner.run_sync(
        ai.Agent(name="writer", model="openai:gpt-4o", tools=[ai.agent_tool(subagent, "research")]),
        "Write a brief report",
        client=client,
        trace_sinks=[sink],
    )
    runs = JsonlTraceStore(sink.path).list_runs()
    assert any(r["parent_run_id"] == parent.trace_id for r in runs)

Enforcing Tool Policies

Tool policies control whether specific function calls execute or are blocked. The test file tests/agents/test_agent_integration_flows.py (lines 73‑106) validates policies such as RequireApprovalPolicy and custom callables.

Tests define policy functions returning ai.ToolPolicyDecision(allowed=False, ...) and attach them to the runner. Assertions confirm that tool.denied events appear in the trace sink when policies block execution.

Async Runner Equivalence Testing

The async and sync execution paths must produce identical results. Tests in tests/agents/test_async_runner.py (lines 12‑40) mock the asynchronous client.chat.completions.acreate with AsyncMock and compare outputs against the synchronous create method.

Tests call both await Runner.run(...) and Runner.run_sync(...) on the same Agent configuration, then assert equality of final_output, messages, and status fields.

import pytest, asyncio
from unittest.mock import AsyncMock, Mock
import aisuite as ai

@pytest.mark.asyncio
async def test_async_vs_sync():
    sync_client = ai.Client()
    sync_client.chat.completions.create = Mock(return_value=chat_response("sync"))
    async_client = ai.Client()
    async_client.chat.completions.acreate = AsyncMock(return_value=chat_response("async"))

    agent = ai.Agent(name="demo", model="openai:gpt-4o")
    sync_res = ai.Runner.run_sync(agent, "ping", client=sync_client)
    async_res = await ai.Runner.run(agent, "ping", client=async_client)

    assert sync_res.final_output == async_res.final_output
    assert sync_res.messages == async_res.messages

Streaming Tool-Call Handling

When stream=True, aisuite yields incremental delta.tool_calls chunks that must be assembled correctly. The test test_audio_transcriptions_create_stream_output in tests/providers/test_openai_provider.py (lines 61‑86) demonstrates this pattern.

Tests mock providers to return lists of delta and done events, consume the async generator from client.chat.completions.create(stream=True), and assert that assembled tool‑call payloads match expectations.

Environment Fixture Patterns

Tests requiring API keys use pytest fixtures to inject dummy credentials. In tests/providers/test_openai_provider.py (lines 19‑23), a set_api_key_env_var fixture with autouse=True sets OPENAI_API_KEY via monkeypatch, ensuring every test runs with the environment variable present without exposing real secrets.

Summary

• Message modeling: Validate Message, ChatCompletionMessageToolCall, and Function objects in aisuite/framework/message.py to ensure provider payload compatibility.
• ASR validation: Test Pydantic models like TranscriptionResult and Segment in tests/framework/test_asr_models.py for schema enforcement.
• Parameter mapping: Use ParamValidator.validate_and_map() in tests/framework/test_asr_params.py to verify provider‑specific key translations and extra_param_mode behaviors.
• Agent tracing: Mock providers and verify JsonlTraceStore output in tests/agents/test_agent_integration_flows.py to confirm parent/child run relationships.
• Policy enforcement: Test tool denial flows by attaching custom policies and checking trace sinks for tool.denied events.
• Async parity: Ensure Runner.run and Runner.run_sync produce identical outputs by mocking both sync and async client methods in tests/agents/test_async_runner.py.
• Environment isolation: Use autouse fixtures with monkeypatch to inject dummy API keys without modifying global state.

Frequently Asked Questions

How do I mock provider responses when testing aisuite agents?

Create a Message object with the expected role and content (or tool_calls), then assign it as the return_value or side_effect of client.providers["provider_name"].chat_completions_create. Pass this mocked client to Runner.run_sync or Runner.run to exercise agent logic without network calls.

What is the purpose of ParamValidator in aisuite testing?

ParamValidator normalizes OpenAI‑style parameter names (like language or temperature) into provider‑specific equivalents (such as language_code for Google). Tests validate that validate_and_map() correctly transforms dictionaries and respects the extra_param_mode setting (strict, warn, or permissive) when encountering unknown keys.

How does aisuite handle async versus sync test execution?

The library provides Runner.run for async and Runner.run_sync for synchronous execution. According to tests/agents/test_async_runner.py, both methods should yield identical final_output and messages when given the same inputs. Tests mock chat.completions.create with Mock and chat.completions.acreate with AsyncMock to verify behavioral parity.

Where are tool policy decisions tested in the aisuite framework?

Tool policy enforcement is tested in tests/agents/test_agent_integration_flows.py (lines 73‑106). These tests attach policies like RequireApprovalPolicy or custom callables returning ToolPolicyDecision objects to a runner, trigger tool calls, and assert that the trace sink captures tool.denied events when policies block execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}