How to Set Up Web-Augmented Literature Search Using Tavily and Google Scholar in AutoResearchClaw

AutoResearchClaw enables real-time literature enrichment by orchestrating Tavily AI search, DuckDuckGo fallback scraping, and Google Scholar integration through the WebSearchAgent class to automatically generate BibTeX entries and searchable research contexts.

AutoResearchClaw (ARC) streamlines systematic literature reviews by fetching current web content and scholarly papers through unified search backends. Setting up web-augmented literature search using Tavily and Google Scholar involves configuring the WebSearchConfig and initializing the WebSearchAgent with your API credentials. This implementation leverages the actual source code from the aiming-lab/AutoResearchClaw repository to transform topic strings into curated research datasets.

Architecture and Core Components

The web search system relies on three primary clients orchestrated by a high-level agent. Understanding these components ensures you can debug, extend, or optimize the literature collection pipeline.

WebSearchClient

The WebSearchClient class in researchclaw/web/search.py serves as the low-level HTTP interface. It implements _search_tavily() for authenticated API calls and _search_duckduckgo() as a free HTML-scraping fallback. The client handles rate-limiting, normalizes responses into WebSearchResponse objects, and automatically degrades to DuckDuckGo when the Tavily API key is absent or requests fail.

GoogleScholarClient

Located in researchclaw/web/scholar.py, the GoogleScholarClient wraps the scholarly Python library to provide search(), get_citations(), and search_author() methods. This client includes built-in rate-limiting to prevent IP blocking when querying Google Scholar's unofficial endpoints.

WebSearchAgent

The WebSearchAgent in researchclaw/web/agent.py orchestrates the full retrieval workflow. According to the source code, this agent:

1. Generates search queries from a topic string
2. Calls WebSearchClient for web results (Tavily primary, DuckDuckGo fallback)
3. Calls GoogleScholarClient if enable_scholar is true
4. Optionally crawls result URLs and extracts PDF text
5. Returns unified WebSearchAgentResult objects

Literature Pipeline Integration

The _literature pipeline stage in researchclaw/pipeline/stage_impls/_literature.py consumes the agent's output. It merges ScholarPaper objects with other candidate sources, saves a searchable Markdown context, and generates a references.bib file for downstream drafting stages.

Configuration Setup

All search parameters are centralized in WebSearchConfig defined in researchclaw/config.py. You can configure the system via YAML configuration files or environment variables.

YAML Configuration

Add the following section to your ARC configuration file:

web_search:
  enabled: true
  tavily_api_key: "tvly-xxxxxxxxxxxxxxxxxxxx"
  tavily_api_key_env: "TAVILY_API_KEY"
  enable_scholar: true
  enable_crawling: false
  enable_pdf_extraction: false
  max_web_results: 10
  max_scholar_results: 10

Environment Variables

For security-sensitive deployments, omit tavily_api_key from the YAML and export the environment variable before running ARC:

export TAVILY_API_KEY="tvly-your-key-here"

The WebSearchConfig class automatically resolves the API key from the environment variable specified in tavily_api_key_env.

Implementation Examples

Standalone Web and Scholar Search

Use the individual clients directly for lightweight scripts or custom preprocessing:

from researchclaw.web.search import WebSearchClient
from researchclaw.web.scholar import GoogleScholarClient

# Tavily search (falls back to DuckDuckGo if key is None)

ws = WebSearchClient(api_key="tvly-REPLACE_WITH_YOUR_KEY")
response = ws.search("knowledge distillation survey 2024", max_results=5)

for r in response.results:
    print(f"[{r.source}] {r.title} → {r.url}")

# Google Scholar search

gs = GoogleScholarClient()
papers = gs.search("knowledge distillation", limit=5)

for p in papers:
    print(f"[Scholar] {p.title} ({p.year}) – citations: {p.citation_count}")

# Retrieve citations for the first paper

if papers:
    citations = gs.get_citations(papers[0].scholar_id, limit=3)
    print("\nCitations of first paper:")
    for c in citations:
        print(f"  • {c.title}")

Using the WebSearchAgent Orchestrator

For full pipeline integration, instantiate the WebSearchAgent with your configuration parameters:

from researchclaw.web.agent import WebSearchAgent

agent = WebSearchAgent(
    tavily_api_key="tvly-REPLACE_WITH_YOUR_KEY",
    enable_scholar=True,
    enable_crawling=False,
    enable_pdf=False,
    max_web_results=8,
    max_scholar_results=8,
)

result = agent.search_and_extract(
    topic="self-supervised learning for graphs",
    search_queries=None,  # Auto-generates queries if None

)

print(f"Web results: {len(result.web_results)}")
print(f"Scholar papers: {len(result.scholar_papers)}")
print(f"Search answer: {result.search_answer}")

The result.to_dict() method serializes the WebSearchAgentResult for storage or downstream processing.

Command-Line Pipeline Execution

Run the complete literature collection stage from the terminal:

researchclaw run --stage literature_collect --config config.yaml

Upon completion, the stage directory contains:

• web_context.md – Excerpted web content formatted for LLM context windows
• web_search_result.json – Raw API responses from Tavily and Scholar
• references.bib – Ready-to-use BibTeX entries for all retrieved papers

Summary

• WebSearchClient in researchclaw/web/search.py handles Tavily API calls with automatic DuckDuckGo fallback when credentials are missing.
• GoogleScholarClient in researchclaw/web/scholar.py provides rate-limited access to Google Scholar via the scholarly library.
• WebSearchAgent in researchclaw/web/agent.py unifies web and scholarly search with optional crawling and PDF extraction.
• Configuration occurs through WebSearchConfig in researchclaw/config.py, supporting both YAML files and environment variables like TAVILY_API_KEY.
• The _literature pipeline stage automatically converts search results into BibTeX and Markdown context files for downstream processing.

Frequently Asked Questions

What happens if I don't provide a Tavily API key?

AutoResearchClaw automatically falls back to DuckDuckGo HTML scraping via the _search_duckduckgo() method in researchclaw/web/search.py. While this requires no API key, it may have lower reliability and rate limits compared to the Tavily service.

How does AutoResearchClaw handle rate limiting with Google Scholar?

The GoogleScholarClient class implements built-in delays and retry logic around the scholarly library calls to prevent IP blocking. However, for large-scale searches, you should still implement additional throttling or proxy rotation, as Google Scholar does not provide an official API.

Can I disable web crawling and use only the search APIs?

Yes. Set enable_crawling: false and enable_pdf_extraction: false in your WebSearchConfig. This configuration uses only the Tavily/DuckDuckGo search APIs and Google Scholar metadata without downloading full web pages or PDFs, significantly reducing execution time and HTTP traffic.

Where are the search results stored after a pipeline run?

The _literature stage in researchclaw/pipeline/stage_impls/_literature.py writes three files to the stage output directory: web_context.md containing excerpted content for LLM prompts, web_search_result.json with raw API responses, and references.bib containing formatted citations for all retrieved scholarly papers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}