How FreeLLMAPI Supports Structured Outputs Across Different LLM Providers

FreeLLMAPI implements a provider-agnostic structured output pipeline that automatically detects model capabilities, enforces JSON validity, and routes requests exclusively to providers capable of emitting machine-readable formats.

FreeLLMAPI is an open-source unified API gateway designed to normalize access across heterogeneous large language model providers. When handling structured outputs—such as JSON objects or tool calls—the framework applies a multi-layer normalization strategy that guarantees consistent, valid responses regardless of whether the underlying provider natively supports structured formatting.

Provider Capability Detection

FreeLLMAPI maintains a capability registry that explicitly flags which models support structured tool calls. This prevents the router from selecting incompatible providers when structured output is required.

Capability Flags in the Provider Index

In server/src/providers/index.ts, each model definition includes a capabilities object that declares structured tool call support. For example, models like gpt-oss-120b are explicitly marked with flags indicating their ability to emit structured responses. When the router evaluates candidates, it inspects m.capabilities?.structuredToolCalls to determine eligibility for structured output requests.

// Conceptual example based on provider index structure
const modelCapabilities = {
  "gpt-oss-120b": {
    structuredToolCalls: true,  // Indicates native JSON/tool call support
    // additional capability flags...
  }
};

Request Routing and Model Filtering

The central routing logic actively filters the provider pool based on structured output requirements, ensuring that only capable models receive traffic for JSON-constrained requests.

The requireTools Filter

Inside server/src/services/router.ts (around line 1095), the router checks the requireTools flag on incoming requests. When this flag is set, the system excludes any model lacking structured tool call capabilities:

if (request.requireTools) {
  // Filter to only providers supporting structured tool_calls
  candidates = candidates.filter(m => m.capabilities?.structuredToolCalls);
}

This filtering mechanism operates before any upstream requests are dispatched, preventing wasted calls to providers that would return unstructured text.

JSON Enforcement and Healing

Even when providers claim structured output support, responses may contain markdown fences, explanatory prose, or malformed JSON. FreeLLMAPI implements a strict enforcement layer to sanitize these outputs.

The enforceJsonContent Function

Located in server/src/lib/structured-output.ts, the enforceJsonContent function processes every provider response to guarantee valid JSON. The function executes a multi-stage parsing strategy:

• Whitespace trimming and immediate JSON parsing
• Markdown fence detection: Extracts JSON wrapped in ```json blocks
• Substring extraction: Searches for the longest JSON-like substring within prose explanations
• Retry classification: Returns a retryable failure status if no valid JSON is found, triggering the router to fall back to an alternative model

// Example usage within the response pipeline
import { enforceJsonContent } from "./lib/structured-output";

function processProviderResponse(rawText: string) {
  const result = enforceJsonContent(rawText);
  
  if (!result.ok) {
    // Signals router to attempt next candidate model
    throw new RetryableError("Structured output validation failed");
  }
  
  return result.content; // Guaranteed valid JSON
}

Tool Call Normalization

Some providers—particularly Anthropic and Google—return tool calls as free-form text rather than structured objects. FreeLLMAPI normalizes these divergent formats through a dedicated rescue module.

Rescuing Inline Tool Calls

The server/src/lib/tool-call-rescue.ts module parses inline tool call dialects embedded in text responses. It converts these inline representations into proper structured tool_calls objects before the response reaches the client. This ensures that upstream consumers receive a uniform data shape regardless of whether the provider natively supports structured tool calls or requires text parsing.

Response Format Forwarding

When clients request explicit JSON output via response_format: {type: "json_object"}, FreeLLMAPI forwards this directive unchanged to the upstream provider. However, the framework does not rely solely on the provider's compliance. Instead, the enforceJsonContent validation runs on all returning payloads, creating a safety net that catches providers that ignore or mishandle the formatting directive.

// Client request example
const response = await fetch("https://api.freellmapi.com/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "openrouter/gpt-4o-mini",
    messages: [{ role: "user", content: "Provide weather data as JSON" }],
    response_format: { type: "json_object" }  // Forwarded to provider
  })
});

Error Classification for Structured Outputs

FreeLLMAPI differentiates between transient formatting errors and permanent failures using structured error analysis. In server/src/lib/error-classify.ts, the system gives precedence to structured error status codes (such as 401 responses carrying structured error payloads) over generic text errors. This distinction enables the router to make intelligent retry decisions—attempting alternative providers for retryable structured output failures while surfacing authentication or permission errors immediately to the client.

Summary

• Provider capabilities are explicitly declared in server/src/providers/index.ts using the structuredToolCalls flag, enabling proactive filtering of incompatible models.
• Router logic in server/src/services/router.ts excludes non-structured providers when requireTools is enabled, preventing invalid responses before they occur.
• JSON enforcement via server/src/lib/structured-output.ts sanitizes markdown-wrapped JSON, extracts valid substrings from prose, and triggers model fallback on validation failure.
• Tool call rescue in server/src/lib/tool-call-rescue.ts normalizes free-form tool call text from providers like Anthropic and Google into standard structured objects.
• Error classification prioritizes structured error payloads to optimize retry logic and reduce latency for end users.

Frequently Asked Questions

How does FreeLLMAPI handle providers that return JSON inside markdown code blocks?

The enforceJsonContent function in server/src/lib/structured-output.ts specifically detects markdown fences (```json) and extracts the inner JSON content before validation. If the extraction succeeds, the cleaned JSON is returned to the client; if it fails, the request is marked for retry with a different model.

What happens if no available provider supports structured tool calls for a specific model request?

When the requireTools flag is set and no candidates possess the structuredToolCalls capability, the router in server/src/services/router.ts filters the candidate pool to zero, resulting in an immediate error response to the client indicating that no suitable provider is available for the structured output requirement.

Does FreeLLMAPI modify the response_format parameter when forwarding to providers?

No, FreeLLMAPI forwards the response_format parameter—including json_object specifications—unchanged to the upstream provider. However, it applies additional validation layers upon receipt to ensure compliance, creating a safety net for providers that partially ignore or incorrectly implement the formatting directive.

Which providers require the tool call rescue functionality?

According to the source analysis, providers such as Anthropic and Google frequently return tool calls as free-form text rather than structured JSON objects. The server/src/lib/tool-call-rescue.ts module specifically targets these inline dialects, parsing and converting them to standardized tool_calls structures before delivery to the client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}