How the AiSdkBackend Handles the Model/Tool Loop in Apache Maka

The AiSdkBackend drives a continuous AI model → tool → result → model interaction by creating isolated AiSdkTurn instances that stream provider responses, execute host-provided tools via ToolRuntime, and retry failed requests with exponential back-off until completion.

The Apache Maka runtime implements a robust model/tool loop that enables AI agents to iteratively invoke external functions and incorporate results until task completion. The AiSdkBackend class serves as the primary orchestration layer, managing session state and delegating turn execution to specialized handler classes according to the source code in packages/runtime/src/ai-sdk-backend.ts.

The Model/Tool Loop Architecture

The loop follows a strict turn-based concurrency model where each send() invocation creates a fresh AiSdkTurn instance. This isolation ensures that concurrent requests operate independently while sharing the underlying session context and message history.

Turn Initialization via send()

The entry point AiSdkBackend.send() manages active turn tracking and lifecycle coordination:

async *send(input: BackendSendInput): AsyncIterable<SessionEvent> {
  const turn = this.openTurnScope(input);   // ← creates AiSdkTurn
  try {
    yield* turn.run();                       // ← runs the model/tool loop
  } finally {
    this.activeTurns.delete(turn);
    await turn.close();                      // clean up per‑turn resources
  }
}

The backend maintains a Set<AiSdkTurn> at this.activeTurns (lines 274‑285) to support concurrent execution and coordinated stopping across multiple turns.

Core Loop Execution in AiSdkTurn.run()

The actual model/tool loop resides in packages/runtime/src/ai-sdk-turn.ts. The run() method delegates to runWithinScope(), which implements the following sequence:

1. Model Resolution and Request Building

Before streaming begins, the turn resolves the target model and projects runtime events into provider-compatible messages:

• Model resolution: this.deps.modelAdapter.resolveModel() (line ~645) validates model availability and configuration.
• Message projection: this.deps.messageProjection.buildCurrentUserContent() (line ~1069) composes the system prompt, prior conversation history, and current user input into a ModelMessage[] array suitable for the LLM provider.

2. Streaming with Timeout Enforcement

To prevent hanging connections, a StreamWatchdog enforces both connection and idle timeouts:

new StreamWatchdog({...})  // line ~1277

The watchdog aborts the provider request if the stream stalls beyond configured limits. The actual streaming occurs via:

await model.stream(request, providerRequestAbortController.signal, ...)  // line ~1410

3. Tool Invocation and Result Feedback

When the model emits a tool-call token, the per-turn ToolRuntime executes the host-provided implementation:

this.toolRuntime.runToolCall(toolName, args, ...)  // line ~1503

The ToolRuntime (defined in packages/runtime/src/tool-runtime.ts) handles sandbox boundary checks and returns structured ToolResultContent (see providerToolResultContent implementation at lines 222‑272). This result feeds back into the message history as a new provider message, continuing the loop.

4. Step Flushing and Event Emission

After each provider step completes, flushStep() (line ~1008) persists accumulated assistant text and thinking tokens via appendMessage, emitting text_complete and thinking_complete events to the session.

5. Retry Logic with Exponential Back-off

Transient failures—network errors, rate limits, or provider capacity issues—trigger automatic retries:

if (retryable) await this.providerRetrySleep(delay, signal)  // line ~1765

The retry mechanism uses providerRetryDelayMs (configured at lines 960‑966) and respects MAX_PROVIDER_ATTEMPTS_PER_STEP to prevent infinite loops.

6. Telemetry Recording

Throughout execution, the turn records diagnostic data via:

• trace.modelStreamSucceeded(...)
• trace.modelStreamFailed(...) (line ~1702)

These methods capture token usage, pricing estimates, and compaction statistics to ProviderRequestTelemetry.

7. Completion and Cleanup

Upon successful completion or unrecoverable error, the turn pushes a final event:

queue.push({type:'complete', …})  // line ~1820

The stopReason field indicates success, error, or user-initiated termination.

Session Management and Control

Concurrent Turn Tracking

The AiSdkBackend maintains per-turn state (abort controllers, active tools, watchdog instances) inside each AiSdkTurn instance (lines 37‑40 of ai-sdk-turn.ts). This design allows the backend to iterate over this.activeTurns and signal immediate termination via abortController.abort() when AiSdkBackend.stop() is invoked.

Sandbox Boundary Handling

Security boundaries route through respondToSandboxBoundary (lines 61‑68 in ai-sdk-backend.ts), which delegates to the appropriate active turn based on session context.

Optional Memory Compaction

When sessions configure memoryExtraction, the turn triggers AiSdkTurn.dispatchAutomaticMemoryCompaction (line ~668) after successful steps. The AiSdkCompaction class (imported at line 56) handles history truncation while preserving semantic continuity.

Implementation Example

The following example demonstrates a complete interaction using the Apache Maka runtime:

import { AiSdkBackend } from '@maka/runtime';
import { createTestAiSdkBackend } from '@maka/runtime/__tests__/execution-boundary-test-helpers.js';

// 1️⃣  Build a backend (normally injected by the host)
const backend = createTestAiSdkBackend({
  sessionId: 'sess‑123',
  header: {/* …session header… */},
  appendMessage: async msg => {/* persist the AssistantMessage */},
  readExecutionBoundary: async () => {/* read sandbox boundary */},
  tools: [
    {
      name: 'search',
      description: 'Web‑search for a query string.',
      parameters: { type: 'object', properties: { query: {type: 'string'} }, required: ['query'] },
      impl: async ({ query }) => {/* perform search and return JSON */},
    },
  ],
});

// 2️⃣  Send a user request – the backend creates a turn and runs the loop.
(async () => {
  const events = backend.send({
    turnId: 'turn‑1',
    input: {
      text: 'What are the latest headlines about space tourism?',
    },
  });

  for await (const ev of events) {
    if (ev.type === 'text_complete') {
      console.log('Assistant reply:', ev.text);
    } else if (ev.type === 'tool_start') {
      console.log('Tool invoked:', ev.toolName);
    } else if (ev.type === 'tool_result') {
      console.log('Tool result:', ev.result);
    }
  }
})();

This flow illustrates the backend creating a turn, streaming provider events, invoking the search tool, and finally emitting the assistant's synthesized reply.

Summary

• Turn Isolation: Each send() creates a fresh AiSdkTurn with independent state, enabling safe concurrent processing within a single session.
• Resilient Streaming: The StreamWatchdog enforces timeouts during model streaming, while exponential back-off handles transient provider failures.
• Tool Integration: ToolRuntime executes host-provided functions within sandbox boundaries, returning structured results that feed back into the model context.
• Lifecycle Management: The backend tracks active turns in this.activeTurns, enabling coordinated shutdown and resource cleanup via turn.close().
• Observability: Built-in telemetry captures token usage, pricing, and compaction metrics at each step of the loop.

Frequently Asked Questions

How does AiSdkBackend handle multiple concurrent turns?

The backend stores each active turn in a Set<AiSdkTurn> called this.activeTurns (lines 274‑285 of ai-sdk-backend.ts). This allows concurrent send() calls to execute simultaneously while enabling coordinated stopping—when stop() is invoked, the backend iterates over all active turns and signals their abortController instances to terminate immediately.

What happens when a tool call exceeds the sandbox boundary?

The ToolRuntime (in packages/runtime/src/tool-runtime.ts) validates execution boundaries before invoking host-provided implementations. If a boundary violation occurs, the runtime returns an error result encoded as ToolResultContent, which the model receives as a failure message. The AiSdkBackend routes boundary decisions through respondToSandboxBoundary (lines 61‑68) to handle user consent flows when required.

How does the retry mechanism protect against provider rate limits?

The AiSdkTurn implements exponential back-off via providerRetrySleep(delay, signal) (line ~1765), using the configurable providerRetryDelayMs parameter (lines 960‑966). The system respects MAX_PROVIDER_ATTEMPTS_PER_STEP to prevent infinite retries, and only retries explicitly marked retryable errors such as network timeouts or 429 rate-limit responses.

Can developers customize the message projection logic?

Yes. The AiSdkMessageProjection class (in packages/runtime/src/ai-sdk-message-projection.ts) converts runtime events into provider-compatible formats. Developers can extend or replace this component to implement custom serialization for specific LLM providers, enabling support for proprietary message formats or custom system prompt injection strategies while maintaining the canonical event log required by the loop.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}