How OmniRoute Fusion Routing Uses Model Panels and a Judge for LLM Consensus

OmniRoute Fusion routing combines parallel model execution with a judge model to synthesize authoritative answers from multiple LLM perspectives.

OmniRoute's fusion combo strategy implements a two-stage pipeline that distributes prompts across a panel of models, then delegates synthesis to a judge. This architecture balances latency, reliability, and answer quality by capping wait times for slow responders while still capturing diverse model outputs. According to the OmniRoute source code in open-sse/services/fusion.ts, the entire workflow is orchestrated through configurable timeouts, quorum rules, and graceful degradation paths.

Stage 1: Panel Fan-Out for Parallel Execution

The fusion handler begins by broadcasting the incoming request to all models in the configured panel. This parallel dispatch is controlled by several tunable parameters that prevent resource exhaustion and bound latency.

Non-Streaming, Tool-Stripped Panel Calls

Each panel invocation is forced into a non-streaming mode with tools removed:

• The tools and tool_choice parameters are stripped so panel models return plain prose responses
• Streaming is disabled to simplify response collection and timing
• Results are extracted via extractPanelText, which uses extractTextContent from open-sse/translator/helpers/geminiHelper.ts

This design ensures panel responses are uniformly comparable for the downstream judge.

Quorum-Grace Collection with collectPanel

The panel gathering logic implements a quorum-grace mechanism to optimize latency:

Parameter	Default	Purpose
minPanel	2	Minimum successful responses before considering early return
stragglerGraceMs	5000ms	Wait time after quorum for slower models to finish
panelHardTimeoutMs	30000ms	Absolute deadline for any single panel call
maxPanel	40	Maximum panel size to prevent OOM crashes

Once minPanel answers arrive, the grace timer starts. When it expires—or the hard timeout fires—collectPanel returns whatever responses have accumulated. This caps straggler penalty without abandoning complete panels when all models respond quickly.

Oversized panels are rejected at the validation layer: maxPanel defaults to 40 and can be tuned lower for memory-constrained deployments.

Stage 2: Judge Synthesis for Consensus Answers

After panel collection completes, the judge model produces the final response through structured prompt engineering and configurable model selection.

Judge Model Selection Logic

The judge is determined by this priority order:

1. Explicit judgeModel in combo configuration — used if specified
2. panel[0] — the first panel model serves as default judge

This fallback ensures fusion combos work without extra configuration while allowing optimization for specific synthesis tasks.

The Judge Prompt: Anonymized Consensus Analysis

The judge receives a synthesized request containing:

• The original user turn (preserved from the incoming request)
• A judge prompt built by buildJudgePrompt, which:
  • Labels each panel source as "Source N" (anonymized to prevent bias toward model brands)
  • Instructs analysis of consensus, contradictions, partial coverage, unique insights, and blind spots
  • Requires one authoritative final answer

The judge call retains original streaming and tool configurations, so the final response can stream to clients or invoke tools if the original request specified them.

// Example: configuring a fusion combo with custom judge
const myFusionCombo = {
  name: 'my-fusion',
  strategy: 'fusion',
  targets: ['openai/gpt-4', 'anthropic/claude-2', 'google/gemini-1.5'],
  judgeModel: 'openai/gpt-4-judge', // explicit judge overrides panel[0]
  fusionTuning: {
    minPanel: 2,
    stragglerGraceMs: 5000,
    panelHardTimeoutMs: 30000,
    maxPanel: 30,
  },
} as const;

Graceful Degradation Paths

Fusion routing handles partial failures through three distinct paths defined in open-sse/services/fusion.ts:

• Zero successful panel answers → Returns HTTP 503 with per-model failure reasons for debugging
• Exactly one success, no explicit judge → Returns the single answer directly (fast path, no synthesis overhead)
• One success with explicit judge configured → Judge still synthesizes a polished response (quality over speed)

This tiered fallback ensures availability without sacrificing quality when explicitly requested.

Core Implementation Files

File	Role
open-sse/services/fusion.ts	Main handler: handleFusionChat, collectPanel, buildJudgePrompt, extractPanelText
open-sse/services/combo/types.ts	Type definitions: ComboLogger, ResolvedComboTarget
open-sse/utils/error.ts	Error formatting: errorResponse, sanitizeErrorMessage
open-sse/translator/helpers/geminiHelper.ts	Text extraction: extractTextContent
tests/unit/combo-fusion-strategy.test.ts	Validation of fan-out, synthesis, and degradation behaviors

Usage in the Combo Pipeline

import { handleFusionChat } from '@/open-sse/services/fusion';
import type { ComboLogger, ResolvedComboTarget } from '@/open-sse/services/combo/types';

async function routeFusion(
  body: Record<string, unknown>,
  panelModels: (ResolvedComboTarget | string)[],
  log: ComboLogger,
) {
  return handleFusionChat({
    body,
    models: panelModels,
    handleSingleModel: yourSingleModelDispatcher,
    log,
    comboName: 'my-fusion',
    judgeModel: 'openai/gpt-4-judge',
  });
}

The handleSingleModel parameter injects your existing single-model dispatcher, letting fusion reuse your standard routing infrastructure.

Summary

• Panel fan-out executes requests in parallel with non-streaming, tool-stripped calls bounded by panelHardTimeoutMs and maxPanel
• Quorum-grace collection (minPanel + stragglerGraceMs) optimizes latency without sacrificing coverage
• Judge synthesis anonymizes sources as "Source N" and mandates consensus analysis before producing the final answer
• Default judge fallback to panel[0] simplifies configuration; explicit judgeModel overrides when specialized synthesis is needed
• Three degradation paths (503, direct return, or forced synthesis) balance availability and quality

Frequently Asked Questions

How does OmniRoute Fusion prevent slow models from blocking responses?

Fusion uses a quorum-grace timer: once minPanel successful answers arrive, a short stragglerGraceMs timer begins. When it expires—or the panelHardTimeoutMs deadline hits—the collected answers proceed to judgment. This bounds worst-case latency while capturing fast responses immediately.

Can the judge model stream its response or use tools?

Yes. Unlike panel calls, the judge retains the original request's streaming flag and tool definitions. This means the final synthesized answer can stream to clients or invoke tools if the original user request specified them.

What happens if only one panel model succeeds?

Behavior depends on judge configuration. Without an explicit judgeModel, the single answer returns directly for minimal latency. With an explicit judge configured, that model still synthesizes a polished response even from one source—useful when quality guarantees outweigh speed.

Where is the maximum panel size enforced?

The maxPanel limit (default 40) is validated early in handleFusionChat within open-sse/services/fusion.ts. Panels exceeding this threshold are rejected before any network calls begin, preventing memory exhaustion from oversized configurations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}