# How to Set Up Web-Augmented Literature Search Using Tavily and Google Scholar in AutoResearchClaw

> Learn to set up web-augmented literature search with Tavily and Google Scholar in AutoResearchClaw. Enrich your research in real-time and generate BibTeX entries automatically.

- Repository: [AIMING Lab/AutoResearchClaw](https://github.com/aiming-lab/AutoResearchClaw)
- Tags: how-to-guide
- Published: 2026-05-28

---

**AutoResearchClaw enables real-time literature enrichment by orchestrating Tavily AI search, DuckDuckGo fallback scraping, and Google Scholar integration through the `WebSearchAgent` class to automatically generate BibTeX entries and searchable research contexts.**

AutoResearchClaw (ARC) streamlines systematic literature reviews by fetching current web content and scholarly papers through unified search backends. Setting up web-augmented literature search using Tavily and Google Scholar involves configuring the `WebSearchConfig` and initializing the `WebSearchAgent` with your API credentials. This implementation leverages the actual source code from the `aiming-lab/AutoResearchClaw` repository to transform topic strings into curated research datasets.

## Architecture and Core Components

The web search system relies on three primary clients orchestrated by a high-level agent. Understanding these components ensures you can debug, extend, or optimize the literature collection pipeline.

### WebSearchClient

The `WebSearchClient` class in [`researchclaw/web/search.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/web/search.py) serves as the low-level HTTP interface. It implements `_search_tavily()` for authenticated API calls and `_search_duckduckgo()` as a free HTML-scraping fallback. The client handles rate-limiting, normalizes responses into `WebSearchResponse` objects, and automatically degrades to DuckDuckGo when the Tavily API key is absent or requests fail.

### GoogleScholarClient

Located in [`researchclaw/web/scholar.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/web/scholar.py), the `GoogleScholarClient` wraps the `scholarly` Python library to provide `search()`, `get_citations()`, and `search_author()` methods. This client includes built-in rate-limiting to prevent IP blocking when querying Google Scholar's unofficial endpoints.

### WebSearchAgent

The `WebSearchAgent` in [`researchclaw/web/agent.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/web/agent.py) orchestrates the full retrieval workflow. According to the source code, this agent:

1. Generates search queries from a topic string
2. Calls `WebSearchClient` for web results (Tavily primary, DuckDuckGo fallback)
3. Calls `GoogleScholarClient` if `enable_scholar` is true
4. Optionally crawls result URLs and extracts PDF text
5. Returns unified `WebSearchAgentResult` objects

### Literature Pipeline Integration

The `_literature` pipeline stage in [`researchclaw/pipeline/stage_impls/_literature.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/pipeline/stage_impls/_literature.py) consumes the agent's output. It merges `ScholarPaper` objects with other candidate sources, saves a searchable Markdown context, and generates a `references.bib` file for downstream drafting stages.

## Configuration Setup

All search parameters are centralized in `WebSearchConfig` defined in [`researchclaw/config.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/config.py). You can configure the system via YAML configuration files or environment variables.

### YAML Configuration

Add the following section to your ARC configuration file:

```yaml
web_search:
  enabled: true
  tavily_api_key: "tvly-xxxxxxxxxxxxxxxxxxxx"
  tavily_api_key_env: "TAVILY_API_KEY"
  enable_scholar: true
  enable_crawling: false
  enable_pdf_extraction: false
  max_web_results: 10
  max_scholar_results: 10

```

### Environment Variables

For security-sensitive deployments, omit `tavily_api_key` from the YAML and export the environment variable before running ARC:

```bash
export TAVILY_API_KEY="tvly-your-key-here"

```

The `WebSearchConfig` class automatically resolves the API key from the environment variable specified in `tavily_api_key_env`.

## Implementation Examples

### Standalone Web and Scholar Search

Use the individual clients directly for lightweight scripts or custom preprocessing:

```python
from researchclaw.web.search import WebSearchClient
from researchclaw.web.scholar import GoogleScholarClient

# Tavily search (falls back to DuckDuckGo if key is None)

ws = WebSearchClient(api_key="tvly-REPLACE_WITH_YOUR_KEY")
response = ws.search("knowledge distillation survey 2024", max_results=5)

for r in response.results:
    print(f"[{r.source}] {r.title} → {r.url}")

# Google Scholar search

gs = GoogleScholarClient()
papers = gs.search("knowledge distillation", limit=5)

for p in papers:
    print(f"[Scholar] {p.title} ({p.year}) – citations: {p.citation_count}")

# Retrieve citations for the first paper

if papers:
    citations = gs.get_citations(papers[0].scholar_id, limit=3)
    print("\nCitations of first paper:")
    for c in citations:
        print(f"  • {c.title}")

```

### Using the WebSearchAgent Orchestrator

For full pipeline integration, instantiate the `WebSearchAgent` with your configuration parameters:

```python
from researchclaw.web.agent import WebSearchAgent

agent = WebSearchAgent(
    tavily_api_key="tvly-REPLACE_WITH_YOUR_KEY",
    enable_scholar=True,
    enable_crawling=False,
    enable_pdf=False,
    max_web_results=8,
    max_scholar_results=8,
)

result = agent.search_and_extract(
    topic="self-supervised learning for graphs",
    search_queries=None,  # Auto-generates queries if None

)

print(f"Web results: {len(result.web_results)}")
print(f"Scholar papers: {len(result.scholar_papers)}")
print(f"Search answer: {result.search_answer}")

```

The `result.to_dict()` method serializes the `WebSearchAgentResult` for storage or downstream processing.

### Command-Line Pipeline Execution

Run the complete literature collection stage from the terminal:

```bash
researchclaw run --stage literature_collect --config config.yaml

```

Upon completion, the stage directory contains:

- [`web_context.md`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/web_context.md) – Excerpted web content formatted for LLM context windows
- [`web_search_result.json`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/web_search_result.json) – Raw API responses from Tavily and Scholar
- `references.bib` – Ready-to-use BibTeX entries for all retrieved papers

## Summary

- **WebSearchClient** in [`researchclaw/web/search.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/web/search.py) handles Tavily API calls with automatic DuckDuckGo fallback when credentials are missing.
- **GoogleScholarClient** in [`researchclaw/web/scholar.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/web/scholar.py) provides rate-limited access to Google Scholar via the `scholarly` library.
- **WebSearchAgent** in [`researchclaw/web/agent.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/web/agent.py) unifies web and scholarly search with optional crawling and PDF extraction.
- Configuration occurs through `WebSearchConfig` in [`researchclaw/config.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/config.py), supporting both YAML files and environment variables like `TAVILY_API_KEY`.
- The `_literature` pipeline stage automatically converts search results into BibTeX and Markdown context files for downstream processing.

## Frequently Asked Questions

### What happens if I don't provide a Tavily API key?

AutoResearchClaw automatically falls back to DuckDuckGo HTML scraping via the `_search_duckduckgo()` method in [`researchclaw/web/search.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/web/search.py). While this requires no API key, it may have lower reliability and rate limits compared to the Tavily service.

### How does AutoResearchClaw handle rate limiting with Google Scholar?

The `GoogleScholarClient` class implements built-in delays and retry logic around the `scholarly` library calls to prevent IP blocking. However, for large-scale searches, you should still implement additional throttling or proxy rotation, as Google Scholar does not provide an official API.

### Can I disable web crawling and use only the search APIs?

Yes. Set `enable_crawling: false` and `enable_pdf_extraction: false` in your `WebSearchConfig`. This configuration uses only the Tavily/DuckDuckGo search APIs and Google Scholar metadata without downloading full web pages or PDFs, significantly reducing execution time and HTTP traffic.

### Where are the search results stored after a pipeline run?

The `_literature` stage in [`researchclaw/pipeline/stage_impls/_literature.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/pipeline/stage_impls/_literature.py) writes three files to the stage output directory: [`web_context.md`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/web_context.md) containing excerpted content for LLM prompts, [`web_search_result.json`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/web_search_result.json) with raw API responses, and `references.bib` containing formatted citations for all retrieved scholarly papers.