How the Craft Agents Sources System Connects to External Data

The Craft Agents sources system connects to external data through the Model-Control-Protocol (MCP) client infrastructure, which abstracts transport details and provides a uniform API for querying HTTP endpoints or local executables at runtime.

The craft-ai-agents/craft-agents-oss repository implements a robust data connection layer that treats every external feed as a first-class object. The Craft Agents sources system enables agents to query remote services dynamically, using a unified configuration format that supports both HTTP APIs and local command-line tools.

Source Configuration Structure

External connections begin with a structured definition. The McpSourceConfig interface in packages/shared/src/sources/types.ts defines the contract that every external source must implement.

The configuration specifies the transport mechanism, authentication method, and connection endpoints:

// packages/shared/src/sources/types.ts
export interface McpSourceConfig {
  id: string;
  name: string;
  slug: string;
  provider: string;          // e.g. "mintlify"
  type: "mcp";
  mcp: {
    transport: "http" | "stdio";
    url?: string;           // for HTTP transport
    command?: string;       // for stdio transport
    authType: "none" | "token" | …;
  };
  // … other UI-related fields
}

Users declare these sources in the workspace-level configuration file at workspace/.craft/sources.json. Each entry maps to a specific external service the agent can query:

{
  "id": "my-weather-api",
  "name": "Weather API",
  "slug": "weather",
  "provider": "open-weather",
  "type": "mcp",
  "mcp": {
    "transport": "http",
    "url": "https://api.openweathermap.org/data/2.5",
    "authType": "token"
  },
  "icon": "☀️",
  "enabled": true
}

Loading and Managing Sources

The source manager handles the lifecycle of external connections. Located in packages/shared/src/agent/core/source-manager.ts, this component reads the workspace's sources.json file and transforms each configuration entry into a LoadedSource object.

When loadSources() is invoked, the manager:

• Parses the workspace configuration
• Resolves each entry to a LoadedSource instance
• Registers active sources with the MCP client pool

// Simplified loading workflow
import { loadSources } from '@/agent/source-manager';
const loaded = await loadSources(workspaceId, workspaceRoot);
// loaded is an array of LoadedSource objects, each containing the MCP client.

Built-in Source Compatibility

The system maintains backward compatibility through built-in source shims. The file packages/shared/src/sources/builtin-sources.ts contains placeholder implementations that return empty lists but demonstrate the MCP source shape. For example, the builtin-craft-agents-docs entry shows the HTTP URL structure expected for documentation sources.

Transport Layer Implementation

The Craft Agents sources system abstracts transport details through the MCP client layer. Each source receives an appropriate client based on its declared transport type.

HTTP Transport

For remote APIs, the system uses HttpMcpClientConfig defined in packages/shared/src/mcp/client.ts. This implementation opens persistent HTTP connections to the URLs specified in source configurations, handling JSON-RPC message serialization over standard POST requests.

The HTTP client implementation follows this pattern:

// HTTP client implementation excerpt
import fetch from 'node-fetch';

export async function httpMcpCall(config: HttpMcpClientConfig, payload: any) {
  const response = await fetch(config.url, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(payload)
  });
  return response.json();
}

Stdio Transport

Local command-line tools connect via StdioMcpClientConfig, also defined in packages/shared/src/mcp/client.ts. This transport spawns the executable specified in the command field, piping JSON-RPC messages through the process's stdin and stdout streams. This enables integration with local databases, file processors, or custom scripts without network overhead.

Querying External Data

When an agent requires external data, it invokes the callTool method from packages/shared/src/mcp/mcp-pool.ts. This centralized pool selects the appropriate transport client for the requested source, forwards the tool invocation, and returns the response as a standard JavaScript value.

The workflow hides transport complexity from the agent core:

// Using the source to fetch data
import { callTool } from '@/mcp/mcp-pool';

const result = await callTool({
  sourceId: 'my-weather-api',
  tool: 'search',
  args: { query: 'London' }
});
// `result` now holds the weather data returned by the remote service.

The source manager simultaneously updates connection status (connected, error, etc.) and caches responses to optimize repeated queries. Error states surface automatically to the user interface, providing visibility into external service availability.

Summary

• Craft Agents sources system treats external data feeds as first-class objects configured through McpSourceConfig in packages/shared/src/sources/types.ts.
• The source manager in packages/shared/src/agent/core/source-manager.ts loads configurations and initializes appropriate MCP clients for each source.
• Transport abstraction occurs through packages/shared/src/mcp/client.ts, supporting both HTTP endpoints and stdio-based local executables.
• Data retrieval happens via callTool in packages/shared/src/mcp/mcp-pool.ts, which routes requests to the correct transport layer and returns standardized responses.
• User configurations reside in workspace/.craft/sources.json, defining provider details, authentication, and transport-specific parameters.

Frequently Asked Questions

What file format defines external sources in Craft Agents?

External sources are defined in JSON format within the workspace configuration file located at workspace/.craft/sources.json. Each entry follows the McpSourceConfig interface structure, specifying the provider, transport type (HTTP or stdio), and authentication credentials required to connect to the external service.

How does Craft Agents handle different types of external connections?

The system uses the MCP client infrastructure to normalize transport differences. For HTTP APIs, it implements HttpMcpClientConfig to manage remote connections, while local executables use StdioMcpClientConfig to spawn processes and communicate via stdin/stdout. Both implementations conform to the same callTool interface in packages/shared/src/mcp/mcp-pool.ts.

Where is the source loading logic implemented in the codebase?

The source loading logic resides in packages/shared/src/agent/core/source-manager.ts. The loadSources() function reads the workspace's source configuration, validates each entry against the McpSourceConfig type, and registers active sources with the MCP client pool, creating the appropriate transport client for each connection type.

Can Craft Agents connect to local command-line tools as data sources?

Yes. The Craft Agents sources system supports local executables through the stdio transport mechanism. When a source configuration specifies "transport": "stdio" and provides a command string, the system spawns the executable and pipes JSON-RPC messages through the process's standard input and output streams, enabling seamless integration with local scripts and tools.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}