How to Integrate External Search Providers (Brave, Baidu, and SearXNG) in OpenMAIC

OpenMAIC unifies web search behind a single searchWeb API that routes requests to provider-specific implementations in lib/web-search/, supporting Brave (API or scrape), Baidu (multi-source), and SearXNG (self-hosted) through a consistent TypeScript interface.

OpenMAIC provides a modular web-search abstraction layer that makes it straightforward to integrate external search providers without modifying core application logic. The architecture centers on a unified dispatcher in lib/web-search/index.ts that delegates to provider-specific modules, each conforming to a standard contract for querying and result formatting.

Understanding the Unified Search Architecture

The entry point for all search operations is the searchWeb function exported from lib/web-search/index.ts. This dispatcher accepts a configuration object that includes a providerId and switches execution to the appropriate implementation based on this identifier (lines 38-74).

Each provider module exports a searchWith<Provider> function adhering to this contract:

export async function searchWith<Provider>(params: {
  query: string;
  apiKey?: string;
  maxResults?: number;
  baseUrl?: string;
  signal?: AbortSignal;
}): Promise<WebSearchResult>

When searchWeb is invoked, it routes to the corresponding provider implementation and returns a standardized WebSearchResult containing an array of WebSearchSource objects with title, url, content, and score properties.

Integrating Brave Search

The Brave Search integration resides in lib/web-search/brave.ts and offers two operational modes: official API access and fallback HTML scraping.

API Mode vs. Scrape Mode

When an apiKey is provided, the system uses searchWithBraveApi to send authenticated GET requests to https://api.search.brave.com/res/v1/web/search. The API key must be passed in the X-Subscription-Token header. Without an API key, searchWithBrave automatically falls back to searchWithBraveScrape, which fetches the public Brave search page and parses the HTML response.

Key Implementation Functions

• buildBraveSearchUrl (lines 25-31): Constructs the search endpoint URL, optionally incorporating a custom baseUrl.
• parseBraveSearchHtml (lines 60-86): Extracts title, url, and snippet from HTML responses while filtering out Brave-owned URLs.
• searchWithBrave (lines 83-98): The exported entry point that selects between API and scrape modes based on credential availability.

Integrating Baidu Search

Located in lib/web-search/baidu.ts, the Baidu provider aggregates three distinct sources: Qianfan web search, Baidu Baike, and Baidu Scholar. The searchWithBaidu function merges results from selected sub-sources into a single WebSearchResult.

Sub-Source Architecture

The integration uses baiduSubSources to control which backends to query:

• fetchWebSearch (lines 58-95): Performs POST requests to the Qianfan endpoint and maps the references array to WebSearchSource objects.
• fetchBaike (lines 105-135): Queries the Baike endpoint, extracting abstracts for encyclopedic entries.
• fetchScholar (lines 143-180): Retrieves academic paper metadata from Baidu Scholar.

The baiduHeaders helper (lines 23-29) injects the bearer token and required custom headers for authentication.

Integrating SearXNG

For organizations running self-hosted search aggregators, lib/web-search/searxng.ts provides SearXNG integration via its JSON API.

Implementation Details

The searchWithSearxng function requires a baseUrl pointing to your SearXNG instance. It constructs requests to the /search endpoint with format=json appended to the query string.

• buildSearxngSearchUrl (lines 20-26): Normalizes the base URL and appends query parameters.
• mapSearxngResult (lines 29-48): Transforms raw SearXNG entries into WebSearchSource objects, handling missing titles and optional score overrides.

The function validates the baseUrl, executes the fetch, logs unmappable results for debugging, and returns the standardized result set.

Implementation Examples

To integrate these providers in your application, import the unified dispatcher and specify the desired provider:

import { searchWeb } from '@/lib/web-search';

// Brave Search with API key
const braveResult = await searchWeb({
  providerId: 'brave',
  query: 'open source AI assistants',
  apiKey: process.env.BRAVE_API_KEY,
  maxResults: 7,
});

// Baidu with multiple sub-sources
const baiduResult = await searchWeb({
  providerId: 'baidu',
  query: 'large language models',
  apiKey: process.env.BAIDU_API_KEY,
  baiduSubSources: { webSearch: true, baike: false, scholar: true },
});

// Self-hosted SearXNG
const searxResult = await searchWeb({
  providerId: 'searxng',
  query: 'vector databases',
  baseUrl: 'https://my.searxng.instance',
  maxResults: 5,
});

All calls return a WebSearchResult containing an array of WebSearchSource objects with title, url, content, and score properties, plus metadata including responseTime.

Summary

• OpenMAIC abstracts web search through the searchWeb dispatcher in lib/web-search/index.ts, routing requests based on the providerId parameter.
• Brave Search supports both authenticated API access (using X-Subscription-Token) and credential-free HTML scraping via lib/web-search/brave.ts.
• Baidu Search combines Qianfan, Baike, and Scholar sources in lib/web-search/baidu.ts, controlled through the baiduSubSources configuration.
• SearXNG integration in lib/web-search/searxng.ts enables self-hosted search aggregation using the JSON API format.
• All providers conform to the same TypeScript contract, returning standardized WebSearchResult objects regardless of the backend.

Frequently Asked Questions

Do I need an API key to use Brave Search in OpenMAIC?

No. While OpenMAIC supports authenticated API access via the X-Subscription-Token header in searchWithBraveApi, the searchWithBrave function automatically falls back to HTML scraping mode when no apiKey is provided. The scrape mode parses public Brave results using parseBraveSearchHtml, though it may be less stable than the official API.

Can I query multiple Baidu sources simultaneously?

Yes. The searchWithBaidu function accepts a baiduSubSources object that lets you enable or disable Qianfan web search, Baidu Baike, and Baidu Scholar in a single request. The results from each enabled source are merged into one unified array of WebSearchSource objects returned in the WebSearchResult.

What is the minimum configuration required for SearXNG integration?

You must provide a baseUrl pointing to your SearXNG instance endpoint. Unlike Brave or Baidu, SearXNG does not require an API key, but the baseUrl parameter is mandatory for searchWithSearxng to construct the correct JSON API request to /search?q={query}&format=json.

How does OpenMAIC handle differences between provider response formats?

Each provider module in lib/web-search/ implements a specific mapping function—such as mapSearxngResult or parseBraveSearchHtml—that normalizes raw API responses into the standard WebSearchSource structure. This abstraction ensures that the searchWeb dispatcher returns consistent data regardless of which provider you use.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}