# How the Craft Agents Sources System Connects to External Data

> Discover how the Craft Agents sources system connects to external data using the Model-Control-Protocol MCP client for seamless HTTP endpoint and local executable querying.

- Repository: [Craft Ai Agents/craft-agents-oss](https://github.com/craft-ai-agents/craft-agents-oss)
- Tags: internals
- Published: 2026-07-04

---

**The Craft Agents sources system connects to external data through the Model-Control-Protocol (MCP) client infrastructure, which abstracts transport details and provides a uniform API for querying HTTP endpoints or local executables at runtime.**

The `craft-ai-agents/craft-agents-oss` repository implements a robust data connection layer that treats every external feed as a first-class object. The Craft Agents sources system enables agents to query remote services dynamically, using a unified configuration format that supports both HTTP APIs and local command-line tools.

## Source Configuration Structure

External connections begin with a structured definition. The `McpSourceConfig` interface in [`packages/shared/src/sources/types.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sources/types.ts) defines the contract that every external source must implement.

The configuration specifies the transport mechanism, authentication method, and connection endpoints:

```typescript
// packages/shared/src/sources/types.ts
export interface McpSourceConfig {
  id: string;
  name: string;
  slug: string;
  provider: string;          // e.g. "mintlify"
  type: "mcp";
  mcp: {
    transport: "http" | "stdio";
    url?: string;           // for HTTP transport
    command?: string;       // for stdio transport
    authType: "none" | "token" | …;
  };
  // … other UI-related fields
}

```

Users declare these sources in the workspace-level configuration file at [`workspace/.craft/sources.json`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/workspace/.craft/sources.json). Each entry maps to a specific external service the agent can query:

```json
{
  "id": "my-weather-api",
  "name": "Weather API",
  "slug": "weather",
  "provider": "open-weather",
  "type": "mcp",
  "mcp": {
    "transport": "http",
    "url": "https://api.openweathermap.org/data/2.5",
    "authType": "token"
  },
  "icon": "☀️",
  "enabled": true
}

```

## Loading and Managing Sources

The **source manager** handles the lifecycle of external connections. Located in [`packages/shared/src/agent/core/source-manager.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/agent/core/source-manager.ts), this component reads the workspace's [`sources.json`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/sources.json) file and transforms each configuration entry into a `LoadedSource` object.

When `loadSources()` is invoked, the manager:

- Parses the workspace configuration
- Resolves each entry to a `LoadedSource` instance
- Registers active sources with the MCP client pool

```typescript
// Simplified loading workflow
import { loadSources } from '@/agent/source-manager';
const loaded = await loadSources(workspaceId, workspaceRoot);
// loaded is an array of LoadedSource objects, each containing the MCP client.

```

### Built-in Source Compatibility

The system maintains backward compatibility through built-in source shims. The file [`packages/shared/src/sources/builtin-sources.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sources/builtin-sources.ts) contains placeholder implementations that return empty lists but demonstrate the MCP source shape. For example, the `builtin-craft-agents-docs` entry shows the HTTP URL structure expected for documentation sources.

## Transport Layer Implementation

The Craft Agents sources system abstracts transport details through the MCP client layer. Each source receives an appropriate client based on its declared transport type.

### HTTP Transport

For remote APIs, the system uses `HttpMcpClientConfig` defined in [`packages/shared/src/mcp/client.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/mcp/client.ts). This implementation opens persistent HTTP connections to the URLs specified in source configurations, handling JSON-RPC message serialization over standard POST requests.

The HTTP client implementation follows this pattern:

```typescript
// HTTP client implementation excerpt
import fetch from 'node-fetch';

export async function httpMcpCall(config: HttpMcpClientConfig, payload: any) {
  const response = await fetch(config.url, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(payload)
  });
  return response.json();
}

```

### Stdio Transport

Local command-line tools connect via `StdioMcpClientConfig`, also defined in [`packages/shared/src/mcp/client.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/mcp/client.ts). This transport spawns the executable specified in the `command` field, piping JSON-RPC messages through the process's stdin and stdout streams. This enables integration with local databases, file processors, or custom scripts without network overhead.

## Querying External Data

When an agent requires external data, it invokes the `callTool` method from [`packages/shared/src/mcp/mcp-pool.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/mcp/mcp-pool.ts). This centralized pool selects the appropriate transport client for the requested source, forwards the tool invocation, and returns the response as a standard JavaScript value.

The workflow hides transport complexity from the agent core:

```typescript
// Using the source to fetch data
import { callTool } from '@/mcp/mcp-pool';

const result = await callTool({
  sourceId: 'my-weather-api',
  tool: 'search',
  args: { query: 'London' }
});
// `result` now holds the weather data returned by the remote service.

```

The source manager simultaneously updates connection status (`connected`, `error`, etc.) and caches responses to optimize repeated queries. Error states surface automatically to the user interface, providing visibility into external service availability.

## Summary

- **Craft Agents sources system** treats external data feeds as first-class objects configured through `McpSourceConfig` in [`packages/shared/src/sources/types.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sources/types.ts).
- The **source manager** in [`packages/shared/src/agent/core/source-manager.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/agent/core/source-manager.ts) loads configurations and initializes appropriate MCP clients for each source.
- **Transport abstraction** occurs through [`packages/shared/src/mcp/client.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/mcp/client.ts), supporting both HTTP endpoints and stdio-based local executables.
- **Data retrieval** happens via `callTool` in [`packages/shared/src/mcp/mcp-pool.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/mcp/mcp-pool.ts), which routes requests to the correct transport layer and returns standardized responses.
- User configurations reside in [`workspace/.craft/sources.json`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/workspace/.craft/sources.json), defining provider details, authentication, and transport-specific parameters.

## Frequently Asked Questions

### What file format defines external sources in Craft Agents?

External sources are defined in JSON format within the workspace configuration file located at [`workspace/.craft/sources.json`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/workspace/.craft/sources.json). Each entry follows the `McpSourceConfig` interface structure, specifying the provider, transport type (HTTP or stdio), and authentication credentials required to connect to the external service.

### How does Craft Agents handle different types of external connections?

The system uses the **MCP client infrastructure** to normalize transport differences. For HTTP APIs, it implements `HttpMcpClientConfig` to manage remote connections, while local executables use `StdioMcpClientConfig` to spawn processes and communicate via stdin/stdout. Both implementations conform to the same `callTool` interface in [`packages/shared/src/mcp/mcp-pool.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/mcp/mcp-pool.ts).

### Where is the source loading logic implemented in the codebase?

The source loading logic resides in [`packages/shared/src/agent/core/source-manager.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/agent/core/source-manager.ts). The `loadSources()` function reads the workspace's source configuration, validates each entry against the `McpSourceConfig` type, and registers active sources with the MCP client pool, creating the appropriate transport client for each connection type.

### Can Craft Agents connect to local command-line tools as data sources?

Yes. The Craft Agents sources system supports local executables through the stdio transport mechanism. When a source configuration specifies `"transport": "stdio"` and provides a `command` string, the system spawns the executable and pipes JSON-RPC messages through the process's standard input and output streams, enabling seamless integration with local scripts and tools.