Integrating Various Search Engine Providers (Google, Bing, Serper) for Agent Research in MetaGPT

MetaGPT unifies Google, Bing, Serper, and other search providers behind a single async interface using a factory-based plugin architecture that isolates provider-specific logic in thin wrapper classes.

MetaGPT is a multi-agent framework designed for autonomous software development and research tasks. Integrating various search engine providers (Google, Bing, Serper) for agent research is essential for gathering real-time web data, and the framework accomplishes this through a centralized configuration system and pluggable search engine wrappers that expose a uniform API regardless of the backend service.

Architecture Overview

The search integration in MetaGPT relies on three core components that separate configuration, provider selection, and execution logic.

SearchConfig – Declarative Configuration

Located in metagpt/configs/search_config.py, the SearchConfig class is a Pydantic model that defines provider credentials, engine type, and default parameters.

# metagpt/configs/search_config.py

class SearchConfig(YamlModel):
    model_config = ConfigDict(extra="allow")
    api_type: SearchEngineType = SearchEngineType.DUCK_DUCK_GO
    api_key: str = ""
    cse_id: str = ""
    search_func: Optional[Callable] = None
    params: dict = Field(default_factory=lambda: {
        "engine": "google", "google_domain": "google.com", "gl": "us", "hl": "en"
    })

Key fields include api_type for selecting the provider, api_key and cse_id for authentication, and params for provider-specific defaults.

SearchEngineType – Provider Enumeration

The SearchEngineType enum in the same file maps string identifiers to provider constants:

• SERPAPI_GOOGLE – SerpAPI service
• SERPER_GOOGLE – Serper (google.serper.dev)
• DIRECT_GOOGLE – Official Google Custom Search API
• BING – Microsoft Bing Search API
• DUCK_DUCK_GO – DuckDuckGo (no API key required)
• CUSTOM_ENGINE – User-defined callable

SearchEngine – Central Factory

The SearchEngine class in metagpt/tools/search_engine.py acts as a factory that instantiates the correct wrapper based on the engine type. It exposes a unified async run() method that all agents use regardless of the underlying provider.

The _process_extra method handles provider selection:

# metagpt/tools/search_engine.py

def _process_extra(self, run_func: Optional[Callable] = None, **kwargs):
    if self.engine == SearchEngineType.SERPAPI_GOOGLE:
        module = "metagpt.tools.search_engine_serpapi"
        run_func = importlib.import_module(module).SerpAPIWrapper(**kwargs).run
    elif self.engine == SearchEngineType.SERPER_GOOGLE:
        module = "metagpt.tools.search_engine_serper"
        run_func = importlib.import_module(module).SerperWrapper(**kwargs).run
    elif self.engine == SearchEngineType.DIRECT_GOOGLE:
        module = "metagpt.tools.search_engine_googleapi"
        run_func = importlib.import_module(module).GoogleAPIWrapper(**kwargs).run
    elif self.engine == SearchEngineType.DUCK_DUCK_GO:
        module = "metagpt.tools.search_engine_ddg"
        run_func = importlib.import_module(module).DDGAPIWrapper(**kwargs).run
    elif self.engine == SearchEngineType.BING:
        module = "metagpt.tools.search_engine_bing"
        run_func = importlib.import_module(module).BingAPIWrapper(**kwargs).run
    else:
        run_func = self.run_func
    self.run_func = run_func

The run() method simply awaits self.run_func(...) and handles exception logging uniformly.

Supported Search Providers

MetaGPT includes thin, async-compatible wrappers for each major search service.

Google Custom Search API

The GoogleAPIWrapper in metagpt/tools/search_engine_googleapi.py uses the official google-api-python-client library. It executes searches via run_in_executor to maintain async compatibility and returns formatted results containing title, link, and snippet.

Serper (google.serper.dev)

The SerperWrapper in metagpt/tools/search_engine_serper.py sends POST requests to https://google.serper.dev/search. It parses the organic results array and extracts title, link, and snippet, normalizing the schema to match other providers.

Bing Search API

The BingAPIWrapper in metagpt/tools/search_engine_bing.py interfaces with the Microsoft Bing v7 REST API. It handles authentication via subscription key and normalizes the response fields (link, title) to ensure consistent output across the factory.

SerpAPI

The SerpAPIWrapper in metagpt/tools/search_engine_serpapi.py queries https://serpapi.com/search using aiohttp. It processes answer boxes, knowledge graphs, and organic results, providing rich structured data when available.

DuckDuckGo

The DDGAPIWrapper in metagpt/tools/search_engine_ddg.py requires no API key. It uses the duckduckgo_search library, running the synchronous client in a background thread via asyncio.to_thread to maintain the async interface.

Configuration and Usage Examples

YAML-Based Configuration

MetaGPT supports loading search configuration from YAML files using the YamlModel base class.

# config/search.yml

api_type: serper
api_key: sk-xxxxxxxxxxxxxxxx
params:
  gl: us
  hl: en
  num: 8

from metagpt.configs.search_config import SearchConfig
from metagpt.tools.search_engine import SearchEngine

config = SearchConfig.from_yaml("config/search.yml")
engine = SearchEngine.from_search_config(config)
results = await engine.run("multi-agent frameworks")

Programmatic Instantiation

For dynamic provider switching, instantiate SearchEngine directly with the desired type and credentials.

from metagpt.tools.search_engine import SearchEngine
from metagpt.configs.search_config import SearchEngineType

# Google Custom Search

google_engine = SearchEngine(
    engine=SearchEngineType.DIRECT_GOOGLE,
    api_key="GOOGLE_API_KEY",
    cse_id="CUSTOM_SEARCH_ENGINE_ID"
)

# Bing

bing_engine = SearchEngine(
    engine=SearchEngineType.BING,
    api_key="BING_SUBSCRIPTION_KEY"
)

# Run queries

google_results = await google_engine.run("LLM agent architecture", max_results=5)
bing_results = await bing_engine.run("LLM agent architecture", max_results=5)

Extending the System with Custom Providers

To add a new search provider to MetaGPT:

1. Create a wrapper class in metagpt/tools/search_engine_custom.py that implements an async run(query, max_results, as_string, **kwargs) method returning str or list[dict].

2. Extend the enum in metagpt/configs/search_config.py:

   class SearchEngineType(Enum):
       # existing values...
   
       NEW_PROVIDER = "new_provider"

3. Wire the factory in metagpt/tools/search_engine.py inside _process_extra:

   elif self.engine == SearchEngineType.NEW_PROVIDER:
       module = "metagpt.tools.search_engine_custom"
       run_func = importlib.import_module(module).CustomWrapper(**kwargs).run

4. Update configuration to use api_type: new_provider and any required credentials.

Summary

• MetaGPT provides a unified asynchronous interface for web search through the SearchEngine factory class in metagpt/tools/search_engine.py.
• Configuration is handled declaratively via SearchConfig in metagpt/configs/search_config.py, supporting YAML files and environment variables.
• Supported providers include Google Custom Search, Serper, SerpAPI, Bing, and DuckDuckGo, each implemented as a thin async wrapper in metagpt/tools/.
• Extensibility follows a clear pattern: extend the SearchEngineType enum, create a wrapper with an async run() method, and register it in the factory's _process_extra method.

Frequently Asked Questions

How do I switch between search providers without changing code?

Use a YAML configuration file. Define the api_type field with the desired provider identifier (e.g., serper, bing, ddg), then load it via SearchConfig.from_yaml(). The SearchEngine factory automatically instantiates the correct wrapper based on this configuration, allowing you to switch providers by changing the YAML file rather than the codebase.

What is the difference between Serper and SerpAPI?

Both are third-party services that scrape Google search results, but they differ in implementation and pricing. Serper (google.serper.dev) is typically used via POST requests to a dedicated endpoint and often offers a simpler pricing model for high-volume usage. SerpAPI (serpapi.com) provides a more comprehensive feature set including knowledge graphs, answer boxes, and support for multiple engines beyond Google. In MetaGPT, both are supported via search_engine_serper.py and search_engine_serpapi.py respectively.

Does MetaGPT support synchronous search calls?

No, the search architecture is designed to be fully asynchronous. All provider wrappers implement an async run() method, and the central SearchEngine.run() is also async. This design prevents blocking the event loop during HTTP I/O operations. If you need to call search from synchronous code, use asyncio.run() or await the call from within an async function.

How do I handle API rate limits and errors?

The SearchEngine.run() method includes error handling that catches exceptions from underlying wrappers and logs them via metagpt.logs.logger. For rate limits, implement retry logic at the application level or use the ignore_errors parameter to gracefully handle failures. Additionally, you can configure proxy settings through the proxy attribute in SearchConfig to distribute requests across different IP addresses if your provider supports it.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}