# Framework Test Patterns for Testing aisuite Applications: A Complete Guide

> Discover framework test patterns for aisuite applications. Learn to validate Pydantic models, map ASR parameters, and verify agent tracing with mocked Runner workflows.

- Repository: [Andrew Ng/aisuite](https://github.com/andrewyng/aisuite)
- Tags: how-to-guide
- Published: 2026-08-04

---

**aisuite applications are tested through a provider-agnostic framework layer that validates Pydantic message models, maps ASR parameters, and verifies agent tracing through mocked `Runner` workflows.**

The aisuite library (`andrewyng/aisuite`) unifies multiple LLM providers behind a common Python interface. Mastering the framework test patterns for testing aisuite applications allows you to validate chat completions, transcription workflows, and multi-agent orchestration without hitting live APIs.

## Testing Unified Message Objects

The foundation of aisuite testing rests on the `Message` class and its related tool-call models defined in [`aisuite/framework/message.py`](https://github.com/andrewyng/aisuite/blob/main/aisuite/framework/message.py) (lines 26‑33). These Pydantic models represent the OpenAI‑style payload that travels between your application and provider clients.

Tests construct `Message` instances with `tool_calls` lists to simulate LLM responses. A typical pattern involves creating a `ChatCompletionMessageToolCall` containing a `Function` object, then passing this message into `Runner.run_sync` or a mock provider to assert that the resulting output contains expected content.

```python
from aisuite.framework.message import Message, ChatCompletionMessageToolCall, Function
import aisuite as ai

def make_tool_call(name, args, call_id):
    return ChatCompletionMessageToolCall(
        id=call_id,
        type="function",
        function=Function(name=name, arguments=args),
    )

def test_tool_call_flow():
    # Mock LLM response that requests a tool

    response_msg = Message(
        role="assistant",
        tool_calls=[make_tool_call("search", '{"query":"AI"}', "c1")]
    )
    # Feed it into the runner (provider is mocked elsewhere)

    result = ai.Runner.run_sync(
        ai.Agent(name="test", model="openai:gpt-4o", tools=[]),
        "dummy",
        client=ai.Client(),
        initial_message=response_msg,
    )
    assert result.final_output is not None

```

## Validating ASR Data Models

Automatic Speech Recognition (ASR) workflows rely on a separate set of Pydantic models in the framework layer. The test suite in [`tests/framework/test_asr_models.py`](https://github.com/andrewyng/aisuite/blob/main/tests/framework/test_asr_models.py) (lines 15‑40) validates `Word`, `Segment`, `Alternative`, `Channel`, and `TranscriptionResult` classes.

Tests instantiate these models with minimal required fields and use `pydantic.ValidationError` to enforce schema compliance. Optional provider‑specific fields such as `confidence` and `speaker` are tested for correct round‑trip serialization.

## Parameter Mapping with ParamValidator

The `ParamValidator` class in [`aisuite/framework/asr_params.py`](https://github.com/andrewyng/aisuite/blob/main/aisuite/framework/asr_params.py) (lines 17‑38) maps common OpenAI‑style parameters (`language`, `prompt`, `temperature`) to provider‑specific equivalents. The corresponding tests in [`tests/framework/test_asr_params.py`](https://github.com/andrewyng/aisuite/blob/main/tests/framework/test_asr_params.py) (lines 13‑70) verify this translation logic.

Tests call `ParamValidator.validate_and_map(provider, params)` with raw dictionaries and assert that returned dicts contain correctly mapped keys. For example, Google translates `language` to `language_code`. The test suite also exercises the three `extra_param_mode` behaviors—`strict`, `warn`, and `permissive`—to ensure invalid parameters are handled according to configuration.

```python
from aisuite.framework.asr_params import ParamValidator

def test_google_language_mapping():
    validator = ParamValidator(extra_param_mode="strict")
    mapped = validator.validate_and_map("google", {"language": "en"})
    assert mapped == {"language_code": "en-US"}   # Google expands 2‑letter codes

```

## Agent Integration and Tracing Flows

End‑to‑end agent testing occurs in [`tests/agents/test_agent_integration_flows.py`](https://github.com/andrewyng/aisuite/blob/main/tests/agents/test_agent_integration_flows.py) (lines 16‑63). These tests orchestrate **Agent → Sub‑Agent** chains using `ai.agent_tool(subagent, name=...)` and verify that the `JsonlTraceStore` captures parent/child run relationships.

Tests mock the provider’s `chat_completions_create` method to return pre‑crafted `Message` objects containing tool calls. After executing `Runner.run_sync`, assertions verify that the trace store contains both runs with correct `parent_run_id` linkages.

```python
import aisuite as ai
from aisuite.tracing import LocalTraceSink, JsonlTraceStore

def test_parent_child_trace(tmp_path):
    client = ai.Client()
    client.providers["openai"] = Mock()
    # ...set side_effect on provider.chat_completions_create as in the repo...

    sink = LocalTraceSink(tmp_path / "trace.jsonl")
    parent = ai.Runner.run_sync(
        ai.Agent(name="writer", model="openai:gpt-4o", tools=[ai.agent_tool(subagent, "research")]),
        "Write a brief report",
        client=client,
        trace_sinks=[sink],
    )
    runs = JsonlTraceStore(sink.path).list_runs()
    assert any(r["parent_run_id"] == parent.trace_id for r in runs)

```

## Enforcing Tool Policies

Tool policies control whether specific function calls execute or are blocked. The test file [`tests/agents/test_agent_integration_flows.py`](https://github.com/andrewyng/aisuite/blob/main/tests/agents/test_agent_integration_flows.py) (lines 73‑106) validates policies such as `RequireApprovalPolicy` and custom callables.

Tests define policy functions returning `ai.ToolPolicyDecision(allowed=False, ...)` and attach them to the runner. Assertions confirm that `tool.denied` events appear in the trace sink when policies block execution.

## Async Runner Equivalence Testing

The async and sync execution paths must produce identical results. Tests in [`tests/agents/test_async_runner.py`](https://github.com/andrewyng/aisuite/blob/main/tests/agents/test_async_runner.py) (lines 12‑40) mock the asynchronous `client.chat.completions.acreate` with `AsyncMock` and compare outputs against the synchronous `create` method.

Tests call both `await Runner.run(...)` and `Runner.run_sync(...)` on the same `Agent` configuration, then assert equality of `final_output`, `messages`, and status fields.

```python
import pytest, asyncio
from unittest.mock import AsyncMock, Mock
import aisuite as ai

@pytest.mark.asyncio
async def test_async_vs_sync():
    sync_client = ai.Client()
    sync_client.chat.completions.create = Mock(return_value=chat_response("sync"))
    async_client = ai.Client()
    async_client.chat.completions.acreate = AsyncMock(return_value=chat_response("async"))

    agent = ai.Agent(name="demo", model="openai:gpt-4o")
    sync_res = ai.Runner.run_sync(agent, "ping", client=sync_client)
    async_res = await ai.Runner.run(agent, "ping", client=async_client)

    assert sync_res.final_output == async_res.final_output
    assert sync_res.messages == async_res.messages

```

## Streaming Tool-Call Handling

When `stream=True`, aisuite yields incremental `delta.tool_calls` chunks that must be assembled correctly. The test `test_audio_transcriptions_create_stream_output` in [`tests/providers/test_openai_provider.py`](https://github.com/andrewyng/aisuite/blob/main/tests/providers/test_openai_provider.py) (lines 61‑86) demonstrates this pattern.

Tests mock providers to return lists of delta and done events, consume the async generator from `client.chat.completions.create(stream=True)`, and assert that assembled tool‑call payloads match expectations.

## Environment Fixture Patterns

Tests requiring API keys use pytest fixtures to inject dummy credentials. In [`tests/providers/test_openai_provider.py`](https://github.com/andrewyng/aisuite/blob/main/tests/providers/test_openai_provider.py) (lines 19‑23), a `set_api_key_env_var` fixture with `autouse=True` sets `OPENAI_API_KEY` via `monkeypatch`, ensuring every test runs with the environment variable present without exposing real secrets.

## Summary

- **Message modeling**: Validate `Message`, `ChatCompletionMessageToolCall`, and `Function` objects in [`aisuite/framework/message.py`](https://github.com/andrewyng/aisuite/blob/main/aisuite/framework/message.py) to ensure provider payload compatibility.
- **ASR validation**: Test Pydantic models like `TranscriptionResult` and `Segment` in [`tests/framework/test_asr_models.py`](https://github.com/andrewyng/aisuite/blob/main/tests/framework/test_asr_models.py) for schema enforcement.
- **Parameter mapping**: Use `ParamValidator.validate_and_map()` in [`tests/framework/test_asr_params.py`](https://github.com/andrewyng/aisuite/blob/main/tests/framework/test_asr_params.py) to verify provider‑specific key translations and `extra_param_mode` behaviors.
- **Agent tracing**: Mock providers and verify `JsonlTraceStore` output in [`tests/agents/test_agent_integration_flows.py`](https://github.com/andrewyng/aisuite/blob/main/tests/agents/test_agent_integration_flows.py) to confirm parent/child run relationships.
- **Policy enforcement**: Test tool denial flows by attaching custom policies and checking trace sinks for `tool.denied` events.
- **Async parity**: Ensure `Runner.run` and `Runner.run_sync` produce identical outputs by mocking both sync and async client methods in [`tests/agents/test_async_runner.py`](https://github.com/andrewyng/aisuite/blob/main/tests/agents/test_async_runner.py).
- **Environment isolation**: Use `autouse` fixtures with `monkeypatch` to inject dummy API keys without modifying global state.

## Frequently Asked Questions

### How do I mock provider responses when testing aisuite agents?

Create a `Message` object with the expected `role` and `content` (or `tool_calls`), then assign it as the `return_value` or `side_effect` of `client.providers["provider_name"].chat_completions_create`. Pass this mocked client to `Runner.run_sync` or `Runner.run` to exercise agent logic without network calls.

### What is the purpose of ParamValidator in aisuite testing?

`ParamValidator` normalizes OpenAI‑style parameter names (like `language` or `temperature`) into provider‑specific equivalents (such as `language_code` for Google). Tests validate that `validate_and_map()` correctly transforms dictionaries and respects the `extra_param_mode` setting (`strict`, `warn`, or `permissive`) when encountering unknown keys.

### How does aisuite handle async versus sync test execution?

The library provides `Runner.run` for async and `Runner.run_sync` for synchronous execution. According to [`tests/agents/test_async_runner.py`](https://github.com/andrewyng/aisuite/blob/main/tests/agents/test_async_runner.py), both methods should yield identical `final_output` and `messages` when given the same inputs. Tests mock `chat.completions.create` with `Mock` and `chat.completions.acreate` with `AsyncMock` to verify behavioral parity.

### Where are tool policy decisions tested in the aisuite framework?

Tool policy enforcement is tested in [`tests/agents/test_agent_integration_flows.py`](https://github.com/andrewyng/aisuite/blob/main/tests/agents/test_agent_integration_flows.py) (lines 73‑106). These tests attach policies like `RequireApprovalPolicy` or custom callables returning `ToolPolicyDecision` objects to a runner, trigger tool calls, and assert that the trace sink captures `tool.denied` events when policies block execution.