How ML Intern Documentation Search Tools Explore Hugging Face Docs

ML Intern provides two agent tools—explore_hf_docs and fetch_hf_docs—that dynamically scrape, index, and retrieve Hugging Face documentation using Whoosh-based full-text search with intelligent caching.

The huggingface/ml-intern repository implements a specialized documentation retrieval system that allows AI agents to navigate the extensive Hugging Face docs corpus programmatically. These ML Intern documentation search tools combine web scraping, in-memory indexing, and relevance ranking to provide structured access to technical documentation without hard-coding knowledge. The implementation resides primarily in agent/tools/docs_tools.py and exposes capabilities through the agent's tool registry in agent/tools/__init__.py.

Architecture of the Documentation Search System

The system employs a two-phase approach: first discovering and indexing available documentation pages, then querying or retrieving specific content. This architecture supports both broad exploration and targeted searches across multiple Hugging Face endpoints.

Endpoint Discovery and Composite Groups

The tools accept an endpoint argument (e.g., transformers, datasets) to scope searches to specific documentation sets. To simplify access to related documentation, the code defines composite endpoints that expand to multiple real endpoints:

COMPOSITE_ENDPOINTS: dict[str, list[str]] = {
    "optimum": ["optimum", "optimum-habana", ...],
    "courses": ["llm-course", "robotics-course", ...],
}

Definition location: lines 26-49 of agent/tools/docs_tools.py.

When processing a request, the system checks this mapping to expand logical groups like optimum into their constituent documentation endpoints automatically.

Web Scraping and Caching Mechanisms

The _fetch_endpoint_docs function scrapes the left-hand navigation sidebar of any Hugging Face docs page (https://huggingface.co/docs/<endpoint>) to build a complete list of available URLs. For each discovered link, it fetches the underlying .md file and extracts metadata including a 200-character glimpse, the page title, and raw Markdown content.

To optimize performance, the implementation maintains module-level cache dictionaries protected by asyncio.Lock:

_docs_cache: dict[str, list[dict[str, str]]] = {}
_index_cache: dict[str, tuple[Any, MultifieldParser]] = {}
_cache_lock = asyncio.Lock()

Cache implementation: lines 55-58 of agent/tools/docs_tools.py.

This caching layer prevents redundant network requests and Whoosh index rebuilds across multiple tool invocations within the same session.

Whoosh Indexing for Full-Text Search

When a user provides a search query, the system constructs an in-memory Whoosh index using RamStorage. The schema indexes three key fields with stemming enabled via StemmingAnalyzer:

schema = Schema(
    title=TEXT(stored=True, analyzer=analyzer),
    content=TEXT(stored=False, analyzer=analyzer),
    glimpse=TEXT(stored=True, analyzer=analyzer),
    ...
)

Schema definition: lines 81-90 of agent/tools/docs_tools.py.

This approach enables fast full-text search across documentation titles and content without persisting indexes to disk.

Query Processing and Result Formatting

The _search_docs function uses a MultifieldParser to search both the title and content fields simultaneously. If query parsing fails, the system implements a fallback mechanism that returns results in default order with an explanatory note.

The _format_results function transforms raw Whoosh hits into human-readable output displaying:

• Page title and URL
• Section identifier
• Relevance score (when query-based)
• 200-character content glimpse

Implementation spans lines 51-81 for formatting and lines 22-25 for query parsing in agent/tools/docs_tools.py.

The explore_hf_docs Handler Workflow

The explore_hf_docs_handler function orchestrates the complete search workflow in agent/tools/docs_tools.py (lines 89-122). The process follows these steps:

1. Input validation: Requires an endpoint parameter and clamps max_results to DEFAULT_MAX_RESULTS (20) or MAX_RESULTS_CAP (50)
2. Authentication: Obtains an HF token from the session for accessing private documentation
3. Document retrieval: Calls _get_docs to fetch or retrieve cached documentation
4. Search execution: Routes to _search_docs if a query exists, otherwise paginates the raw list
5. Response formatting: Returns structured text with optional fallback messages

Gradio Special Case

When endpoint == "gradio", the handler bypasses the standard Whoosh indexing and routes to Gradio-specific endpoints:

• GRADIO_LLMS_TXT_URL (https://gradio.app/llms.txt) for raw document listings
• GRADIO_SEARCH_URL (https://playground-worker.pages.dev/api/prompt) for embedding-based semantic search

Gradio handling logic: lines 100-118 of agent/tools/docs_tools.py.

Fetching Full Documentation Pages with fetch_hf_docs

The hf_docs_fetch_handler provides the second critical capability: retrieving complete Markdown source for specific URLs identified during exploration. This handler:

1. Accepts a url parameter
2. Appends .md extension if missing
3. Authenticates using the session's HF token
4. Returns the raw Markdown content

Implementation: lines 82-104 of agent/tools/docs_tools.py.

Practical Usage Examples

Below are minimal snippets demonstrating the tool workflow within an ML Intern agent context:

import asyncio
from agent.tools.docs_tools import explore_hf_docs_handler, hf_docs_fetch_handler

class MockSession:
    hf_token = "hf_fake_token_123"   # In real use this is a valid token

async def demo():
    # 1️⃣ List the first 5 pages under the `transformers` endpoint that match "pipeline"

    args = {
        "endpoint": "transformers",
        "query": "pipeline",
        "max_results": 5,
    }
    text, ok = await explore_hf_docs_handler(args, session=MockSession())
    print("🗂️ Explore result:", ok)
    print(text)

    # 2️⃣ Grab the full markdown for a specific result URL

    page_url = "https://huggingface.co/docs/transformers/main_classes/pipelines"
    text, ok = await hf_docs_fetch_handler({"url": page_url}, session=MockSession())
    print("\n📄 Full page:", ok)
    print(text[:500])   # show a snippet

asyncio.run(demo())

Typical output includes structured listings with relevance scores (when querying) or paginated results (when browsing), followed by complete Markdown source when fetching specific pages.

Summary

• Dynamic endpoint discovery leverages COMPOSITE_ENDPOINTS to group related documentation sets like optimum and courses into searchable units.
• HTML scraping via _fetch_endpoint_docs extracts navigation sidebars and retrieves raw .md files from huggingface.co/docs/<endpoint>.
• In-memory indexing uses Whoosh with StemmingAnalyzer to enable fast full-text search across titles and content without disk persistence.
• Intelligent caching via _docs_cache and _index_cache prevents redundant network requests and index rebuilds.
• Dual-mode operation supports both keyword-based relevance ranking (using MultifieldParser) and simple pagination for browsing.
• Specialized Gradio handling routes to dedicated endpoints for embedding-based search when accessing Gradio documentation.

Frequently Asked Questions

How does the search ranking work in ML Intern's documentation tools?

The system uses Whoosh, a Python search library, with a MultifieldParser that searches both the title and content fields of indexed documents. The schema employs StemmingAnalyzer to normalize terms, enabling matches between different word forms. Results include relevance scores calculated by Whoosh's scoring algorithm, with higher scores indicating stronger matches in the title or content fields.

What is the difference between explore_hf_docs and fetch_hf_docs?

explore_hf_docs acts as a discovery engine that lists available documentation pages for a given endpoint, optionally ranking them by relevance when a query is provided. It returns titles, URLs, and preview glimpses. In contrast, fetch_hf_docs retrieves the complete raw Markdown source of a specific documentation page URL identified during exploration, enabling the agent to read full technical details.

How does the system handle different Hugging Face documentation endpoints?

The implementation supports dynamic endpoint discovery through the COMPOSITE_ENDPOINTS mapping defined in agent/tools/docs_tools.py. When an agent requests a composite endpoint like optimum, the system automatically expands this to search multiple related endpoints (optimum, optimum-habana, etc.). For each endpoint, the tool scrapes the navigation sidebar of huggingface.co/docs/<endpoint> to build a complete index of available pages.

Why is there special handling for Gradio documentation?

Gradio documentation receives special treatment because it utilizes a distinct architecture from standard Hugging Face docs. When the endpoint is gradio, the tool bypasses the Whoosh indexing pipeline and instead queries Gradio-specific endpoints: llms.txt for raw document listings and a playground worker API for embedding-based semantic search. This ensures the agent can access Gradio's specific documentation format and search capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}