# Integrating Various Search Engine Providers (Google, Bing, Serper) for Agent Research in MetaGPT

> Integrate Google Bing and Serper search providers in MetaGPT using a unified async interface and plugin architecture for efficient agent research. Simplify search logic.

- Repository: [FoundationAgents/MetaGPT](https://github.com/FoundationAgents/MetaGPT)
- Tags: how-to-guide
- Published: 2026-03-04

---

**MetaGPT unifies Google, Bing, Serper, and other search providers behind a single async interface using a factory-based plugin architecture that isolates provider-specific logic in thin wrapper classes.**

MetaGPT is a multi-agent framework designed for autonomous software development and research tasks. Integrating various search engine providers (Google, Bing, Serper) for agent research is essential for gathering real-time web data, and the framework accomplishes this through a centralized configuration system and pluggable search engine wrappers that expose a uniform API regardless of the backend service.

## Architecture Overview

The search integration in MetaGPT relies on three core components that separate configuration, provider selection, and execution logic.

### SearchConfig – Declarative Configuration

Located in [`metagpt/configs/search_config.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/configs/search_config.py), the `SearchConfig` class is a Pydantic model that defines provider credentials, engine type, and default parameters.

```python

# metagpt/configs/search_config.py

class SearchConfig(YamlModel):
    model_config = ConfigDict(extra="allow")
    api_type: SearchEngineType = SearchEngineType.DUCK_DUCK_GO
    api_key: str = ""
    cse_id: str = ""
    search_func: Optional[Callable] = None
    params: dict = Field(default_factory=lambda: {
        "engine": "google", "google_domain": "google.com", "gl": "us", "hl": "en"
    })

```

Key fields include `api_type` for selecting the provider, `api_key` and `cse_id` for authentication, and `params` for provider-specific defaults.

### SearchEngineType – Provider Enumeration

The `SearchEngineType` enum in the same file maps string identifiers to provider constants:

- `SERPAPI_GOOGLE` – SerpAPI service
- `SERPER_GOOGLE` – Serper (google.serper.dev)
- `DIRECT_GOOGLE` – Official Google Custom Search API
- `BING` – Microsoft Bing Search API
- `DUCK_DUCK_GO` – DuckDuckGo (no API key required)
- `CUSTOM_ENGINE` – User-defined callable

### SearchEngine – Central Factory

The `SearchEngine` class in [`metagpt/tools/search_engine.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/tools/search_engine.py) acts as a factory that instantiates the correct wrapper based on the `engine` type. It exposes a unified async `run()` method that all agents use regardless of the underlying provider.

The `_process_extra` method handles provider selection:

```python

# metagpt/tools/search_engine.py

def _process_extra(self, run_func: Optional[Callable] = None, **kwargs):
    if self.engine == SearchEngineType.SERPAPI_GOOGLE:
        module = "metagpt.tools.search_engine_serpapi"
        run_func = importlib.import_module(module).SerpAPIWrapper(**kwargs).run
    elif self.engine == SearchEngineType.SERPER_GOOGLE:
        module = "metagpt.tools.search_engine_serper"
        run_func = importlib.import_module(module).SerperWrapper(**kwargs).run
    elif self.engine == SearchEngineType.DIRECT_GOOGLE:
        module = "metagpt.tools.search_engine_googleapi"
        run_func = importlib.import_module(module).GoogleAPIWrapper(**kwargs).run
    elif self.engine == SearchEngineType.DUCK_DUCK_GO:
        module = "metagpt.tools.search_engine_ddg"
        run_func = importlib.import_module(module).DDGAPIWrapper(**kwargs).run
    elif self.engine == SearchEngineType.BING:
        module = "metagpt.tools.search_engine_bing"
        run_func = importlib.import_module(module).BingAPIWrapper(**kwargs).run
    else:
        run_func = self.run_func
    self.run_func = run_func

```

The `run()` method simply awaits `self.run_func(...)` and handles exception logging uniformly.

## Supported Search Providers

MetaGPT includes thin, async-compatible wrappers for each major search service.

### Google Custom Search API

The `GoogleAPIWrapper` in [`metagpt/tools/search_engine_googleapi.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/tools/search_engine_googleapi.py) uses the official `google-api-python-client` library. It executes searches via `run_in_executor` to maintain async compatibility and returns formatted results containing `title`, `link`, and `snippet`.

### Serper (google.serper.dev)

The `SerperWrapper` in [`metagpt/tools/search_engine_serper.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/tools/search_engine_serper.py) sends POST requests to `https://google.serper.dev/search`. It parses the `organic` results array and extracts `title`, `link`, and `snippet`, normalizing the schema to match other providers.

### Bing Search API

The `BingAPIWrapper` in [`metagpt/tools/search_engine_bing.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/tools/search_engine_bing.py) interfaces with the Microsoft Bing v7 REST API. It handles authentication via subscription key and normalizes the response fields (`link`, `title`) to ensure consistent output across the factory.

### SerpAPI

The `SerpAPIWrapper` in [`metagpt/tools/search_engine_serpapi.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/tools/search_engine_serpapi.py) queries `https://serpapi.com/search` using aiohttp. It processes answer boxes, knowledge graphs, and organic results, providing rich structured data when available.

### DuckDuckGo

The `DDGAPIWrapper` in [`metagpt/tools/search_engine_ddg.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/tools/search_engine_ddg.py) requires no API key. It uses the `duckduckgo_search` library, running the synchronous client in a background thread via `asyncio.to_thread` to maintain the async interface.

## Configuration and Usage Examples

### YAML-Based Configuration

MetaGPT supports loading search configuration from YAML files using the `YamlModel` base class.

```yaml

# config/search.yml

api_type: serper
api_key: sk-xxxxxxxxxxxxxxxx
params:
  gl: us
  hl: en
  num: 8

```

```python
from metagpt.configs.search_config import SearchConfig
from metagpt.tools.search_engine import SearchEngine

config = SearchConfig.from_yaml("config/search.yml")
engine = SearchEngine.from_search_config(config)
results = await engine.run("multi-agent frameworks")

```

### Programmatic Instantiation

For dynamic provider switching, instantiate `SearchEngine` directly with the desired type and credentials.

```python
from metagpt.tools.search_engine import SearchEngine
from metagpt.configs.search_config import SearchEngineType

# Google Custom Search

google_engine = SearchEngine(
    engine=SearchEngineType.DIRECT_GOOGLE,
    api_key="GOOGLE_API_KEY",
    cse_id="CUSTOM_SEARCH_ENGINE_ID"
)

# Bing

bing_engine = SearchEngine(
    engine=SearchEngineType.BING,
    api_key="BING_SUBSCRIPTION_KEY"
)

# Run queries

google_results = await google_engine.run("LLM agent architecture", max_results=5)
bing_results = await bing_engine.run("LLM agent architecture", max_results=5)

```

## Extending the System with Custom Providers

To add a new search provider to MetaGPT:

1. **Create a wrapper class** in [`metagpt/tools/search_engine_custom.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/tools/search_engine_custom.py) that implements an async `run(query, max_results, as_string, **kwargs)` method returning `str` or `list[dict]`.

2. **Extend the enum** in [`metagpt/configs/search_config.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/configs/search_config.py):
   ```python
   class SearchEngineType(Enum):
       # existing values...

       NEW_PROVIDER = "new_provider"
   ```

3. **Wire the factory** in [`metagpt/tools/search_engine.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/tools/search_engine.py) inside `_process_extra`:
   ```python
   elif self.engine == SearchEngineType.NEW_PROVIDER:
       module = "metagpt.tools.search_engine_custom"
       run_func = importlib.import_module(module).CustomWrapper(**kwargs).run
   ```

4. **Update configuration** to use `api_type: new_provider` and any required credentials.

## Summary

- **MetaGPT** provides a unified asynchronous interface for web search through the `SearchEngine` factory class in [`metagpt/tools/search_engine.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/tools/search_engine.py).
- **Configuration** is handled declaratively via `SearchConfig` in [`metagpt/configs/search_config.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/metagpt/configs/search_config.py), supporting YAML files and environment variables.
- **Supported providers** include Google Custom Search, Serper, SerpAPI, Bing, and DuckDuckGo, each implemented as a thin async wrapper in `metagpt/tools/`.
- **Extensibility** follows a clear pattern: extend the `SearchEngineType` enum, create a wrapper with an async `run()` method, and register it in the factory's `_process_extra` method.

## Frequently Asked Questions

### How do I switch between search providers without changing code?

Use a YAML configuration file. Define the `api_type` field with the desired provider identifier (e.g., `serper`, `bing`, `ddg`), then load it via `SearchConfig.from_yaml()`. The `SearchEngine` factory automatically instantiates the correct wrapper based on this configuration, allowing you to switch providers by changing the YAML file rather than the codebase.

### What is the difference between Serper and SerpAPI?

Both are third-party services that scrape Google search results, but they differ in implementation and pricing. **Serper** (`google.serper.dev`) is typically used via POST requests to a dedicated endpoint and often offers a simpler pricing model for high-volume usage. **SerpAPI** (`serpapi.com`) provides a more comprehensive feature set including knowledge graphs, answer boxes, and support for multiple engines beyond Google. In MetaGPT, both are supported via [`search_engine_serper.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/search_engine_serper.py) and [`search_engine_serpapi.py`](https://github.com/FoundationAgents/MetaGPT/blob/main/search_engine_serpapi.py) respectively.

### Does MetaGPT support synchronous search calls?

No, the search architecture is designed to be fully asynchronous. All provider wrappers implement an async `run()` method, and the central `SearchEngine.run()` is also async. This design prevents blocking the event loop during HTTP I/O operations. If you need to call search from synchronous code, use `asyncio.run()` or `await` the call from within an async function.

### How do I handle API rate limits and errors?

The `SearchEngine.run()` method includes error handling that catches exceptions from underlying wrappers and logs them via `metagpt.logs.logger`. For rate limits, implement retry logic at the application level or use the `ignore_errors` parameter to gracefully handle failures. Additionally, you can configure proxy settings through the `proxy` attribute in `SearchConfig` to distribute requests across different IP addresses if your provider supports it.