How OpenWork Manages Inference Provider Provisioning at Scale

OpenWork manages inference provider provisioning at scale through a centralized Den control plane that uses a layered architecture of static model catalogues, tier-based quota systems, and stateless MCP gateways to handle millions of requests across thousands of organizations.

The different-ai/openwork repository implements a sophisticated approach to inference provider provisioning at scale by centralizing all provider management within the Den control plane. This architecture enables organizations to dynamically configure access to upstream models—including OpenRouter, Moonshot, and Tencent—while maintaining strict quota enforcement and horizontal scalability. By separating static configuration from runtime state, OpenWork achieves constant-time lookups for quota checks and supports bulk provisioning operations across enterprise tenants.

The Layered Architecture of the Den Control Plane

OpenWork organizes its inference infrastructure into distinct layers that separate static configuration from dynamic runtime state. This design allows the system to scale horizontally while maintaining strict governance over provider access.

Static Model Catalogue with INFERENCE_MODEL_ALIASES

At the foundation lies the provider catalogue, a static model alias table defined in packages/types/src/den/inference.ts. This table lists every supported upstream model and marks it as enabled with a corresponding usage-factor. Adding a new model requires only a single entry in INFERENCE_MODEL_ALIASES; the table compiles into the client bundle, making the list cheap to read and cache at runtime.

export const INFERENCE_MODEL_ALIASES = {
  "anthropic/claude-3.5-sonnet": {
    upstreamModel: "anthropic/claude-3.5-sonnet",
    displayName: "OpenWork: Claude 3.5 Sonnet",
    enabled: true,
    usageFactor: 1,
  },
} as const;

Tier-Based Quota Management

Organization metadata stores an entity’s tier (tier1 or tier2) and an inference service enabled flag. The tier determines quota limits defined in INFERENCE_TIER_LIMITS, which can be changed in the database without redeploying code. Administrators can upgrade or downgrade organizational capabilities instantly by modifying a single database field.

import {
  INFERENCE_TIER_LIMITS,
  InferenceTier,
  InferenceWindowType,
} from "@openwork/types";

function getQuota(tier: InferenceTier, window: InferenceWindowType): number {
  return INFERENCE_TIER_LIMITS[tier][window];
}

// Example: tier2 gets 1.5B tokens per month
const monthlyQuota = getQuota("tier2", "monthly"); // → 1_500_000_000

Window-Based Rate Limiting and Reset Strategies

The quota system defines three window types (five_hour, weekly, monthly) with corresponding limits per tier. Enforcement relies on constant-time lookups into INFERENCE_TIER_LIMITS, ensuring the service handles millions of requests without performance degradation.

The system also implements configurable reset strategies through INFERENCE_RESET_STRATEGY_BY_WINDOW_TYPE, which determines whether a window resets on a fixed schedule (anchored) or after activity (activity_based). This mapping enables O(1) reset logic and allows the engine to batch resets efficiently across thousands of organizations.

Database-Driven Provider Profiles

Persistent state resides in Den Provider Profiles stored within the Den database (see ee/packages/den-db/src/schema/provider_profiles.ts). These profiles contain per-provider credentials, usage counters, and ACLs. Because profiles exist as ordinary database rows, the Den API can query, update, or delete them in bulk, enabling organization-wide provisioning or revocation in a single transaction.

When an organization signs up for a new provider, the Den UI writes a Provider Profile—containing the provider ID, OAuth tokens, and quota limits—into the database. The system subsequently reads this profile on every inference request to enforce tenant-specific policies.

import { createProviderProfile } from "@openwork/den-api";

await createProviderProfile({
  orgId: "org_123",
  providerId: "openrouter",
  displayName: "OpenRouter",
  credentials: { apiKey: "••••••" },
  tier: "tier1",
});

Stateless MCP Gateway for Request Orchestration

All client calls to inference providers flow through the MCP (Model-Control-Plane) gateway implemented in packages/enterprise-mcp-mock-server/src/protocol/mcp-handler.ts. This gateway adds tenant-aware throttling, quota checks, and error normalization. Because the MCP sits between the client and upstream provider, the same gateway logic scales across all providers and organizations without touching external APIs.

If a quota is exceeded, the MCP returns a standard PROVIDER_THROTTLED error, which clients handle uniformly regardless of the upstream provider. This abstraction layer ensures that quota enforcement remains consistent even as the system adds new model providers.

Scaling Mechanisms and Performance Characteristics

The architecture achieves horizontal scalability through three core mechanisms:

• Stateless MCP instances run behind a load balancer with every request remaining independent because all state lives in the Den DB.
• Fast constant-time lookups for provider configuration, tier limits, and window resets via INFERENCE_TIER_LIMITS and INFERENCE_RESET_STRATEGY_BY_WINDOW_TYPE eliminate database bottlenecks during high-traffic periods.
• Batch provisioning capabilities allow admin tools to create or update many provider profiles in a single database transaction, instantly applying new limits to every running client without service interruption.

Summary

• Centralized control plane: The Den architecture consolidates provider management, enabling single-point provisioning for thousands of organizations.
• Static configuration layer: INFERENCE_MODEL_ALIASES and tier limit tables compile into the client bundle for high-performance lookups.
• Database persistence: Provider profiles in ee/packages/den-db store credentials and ACLs, supporting bulk updates via standard SQL transactions.
• Stateless gateway: The MCP layer enforces quotas and normalizes errors without maintaining request state, allowing horizontal scaling behind load balancers.
• Constant-time enforcement: Window-based quotas and reset strategies use O(1) map lookups to handle millions of requests efficiently.

Frequently Asked Questions

How does OpenWork add new inference providers without system redeployment?

OpenWork adds new providers by updating the INFERENCE_MODEL_ALIASES constant in packages/types/src/den/inference.ts and rebuilding the client bundle. Because this table compiles into static configuration, the system does not require database migrations or service restarts to recognize new upstream models. The compiled bundle caches the provider list, making runtime lookups inexpensive.

What error response does OpenWork return when quota limits are exceeded?

When an organization exceeds its quota for a specific window type, the MCP gateway returns a PROVIDER_THROTTLED error. This standardized error code—implemented in packages/enterprise-mcp-mock-server/src/protocol/mcp-handler.ts—allows client applications to handle rate limiting uniformly across all inference providers without implementing provider-specific error parsing.

How does the architecture ensure horizontal scalability of the MCP gateway?

The MCP gateway maintains statelessness by storing all tenant configuration, credentials, and usage counters in the Den database rather than in memory. Each request carries the necessary context to perform quota checks independently, allowing operators to deploy multiple MCP instances behind a load balancer. This design eliminates session affinity requirements and enables elastic scaling based on request volume.

What determines whether an organization uses tier1 or tier2 quota limits?

Organizations are assigned to tier1 or tier2 through the tier field in their organization metadata stored in the Den database. The INFERENCE_TIER_LIMITS map in packages/types/src/den/inference.ts defines specific token limits for each window type (five_hour, weekly, monthly) per tier. Administrators can change an organization's tier via the Den UI or API without modifying application code, and the MCP gateway applies new limits immediately on subsequent requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}