# How ML Intern Documentation Search Tools Explore Hugging Face Docs

> Discover how ML Intern's documentation search tools navigate Hugging Face docs. Learn about dynamic scraping, indexing, and full-text search with intelligent caching for efficient retrieval.

- Repository: [Hugging Face/ml-intern](https://github.com/huggingface/ml-intern)
- Tags: how-to-guide
- Published: 2026-04-24

---

**ML Intern provides two agent tools—`explore_hf_docs` and `fetch_hf_docs`—that dynamically scrape, index, and retrieve Hugging Face documentation using Whoosh-based full-text search with intelligent caching.**

The `huggingface/ml-intern` repository implements a specialized documentation retrieval system that allows AI agents to navigate the extensive Hugging Face docs corpus programmatically. These **ML Intern documentation search tools** combine web scraping, in-memory indexing, and relevance ranking to provide structured access to technical documentation without hard-coding knowledge. The implementation resides primarily in [`agent/tools/docs_tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/docs_tools.py) and exposes capabilities through the agent's tool registry in [`agent/tools/__init__.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/__init__.py).

## Architecture of the Documentation Search System

The system employs a two-phase approach: first discovering and indexing available documentation pages, then querying or retrieving specific content. This architecture supports both broad exploration and targeted searches across multiple Hugging Face endpoints.

### Endpoint Discovery and Composite Groups

The tools accept an `endpoint` argument (e.g., `transformers`, `datasets`) to scope searches to specific documentation sets. To simplify access to related documentation, the code defines composite endpoints that expand to multiple real endpoints:

```python
COMPOSITE_ENDPOINTS: dict[str, list[str]] = {
    "optimum": ["optimum", "optimum-habana", ...],
    "courses": ["llm-course", "robotics-course", ...],
}

```

*Definition location: lines 26-49 of [`agent/tools/docs_tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/docs_tools.py).*

When processing a request, the system checks this mapping to expand logical groups like `optimum` into their constituent documentation endpoints automatically.

### Web Scraping and Caching Mechanisms

The `_fetch_endpoint_docs` function scrapes the left-hand navigation sidebar of any Hugging Face docs page (`https://huggingface.co/docs/<endpoint>`) to build a complete list of available URLs. For each discovered link, it fetches the underlying `.md` file and extracts metadata including a 200-character **glimpse**, the page title, and raw Markdown content.

To optimize performance, the implementation maintains module-level cache dictionaries protected by `asyncio.Lock`:

```python
_docs_cache: dict[str, list[dict[str, str]]] = {}
_index_cache: dict[str, tuple[Any, MultifieldParser]] = {}
_cache_lock = asyncio.Lock()

```

*Cache implementation: lines 55-58 of [`agent/tools/docs_tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/docs_tools.py).*

This caching layer prevents redundant network requests and Whoosh index rebuilds across multiple tool invocations within the same session.

### Whoosh Indexing for Full-Text Search

When a user provides a search query, the system constructs an in-memory Whoosh index using `RamStorage`. The schema indexes three key fields with stemming enabled via `StemmingAnalyzer`:

```python
schema = Schema(
    title=TEXT(stored=True, analyzer=analyzer),
    content=TEXT(stored=False, analyzer=analyzer),
    glimpse=TEXT(stored=True, analyzer=analyzer),
    ...
)

```

*Schema definition: lines 81-90 of [`agent/tools/docs_tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/docs_tools.py).*

This approach enables fast full-text search across documentation titles and content without persisting indexes to disk.

### Query Processing and Result Formatting

The `_search_docs` function uses a `MultifieldParser` to search both the `title` and `content` fields simultaneously. If query parsing fails, the system implements a fallback mechanism that returns results in default order with an explanatory note.

The `_format_results` function transforms raw Whoosh hits into human-readable output displaying:
- Page title and URL
- Section identifier
- Relevance score (when query-based)
- 200-character content glimpse

*Implementation spans lines 51-81 for formatting and lines 22-25 for query parsing in [`agent/tools/docs_tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/docs_tools.py).*

## The explore_hf_docs Handler Workflow

The `explore_hf_docs_handler` function orchestrates the complete search workflow in [`agent/tools/docs_tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/docs_tools.py) (lines 89-122). The process follows these steps:

1. **Input validation**: Requires an endpoint parameter and clamps `max_results` to `DEFAULT_MAX_RESULTS` (20) or `MAX_RESULTS_CAP` (50)
2. **Authentication**: Obtains an HF token from the session for accessing private documentation
3. **Document retrieval**: Calls `_get_docs` to fetch or retrieve cached documentation
4. **Search execution**: Routes to `_search_docs` if a query exists, otherwise paginates the raw list
5. **Response formatting**: Returns structured text with optional fallback messages

### Gradio Special Case

When `endpoint == "gradio"`, the handler bypasses the standard Whoosh indexing and routes to Gradio-specific endpoints:

- `GRADIO_LLMS_TXT_URL` (`https://gradio.app/llms.txt`) for raw document listings
- `GRADIO_SEARCH_URL` (`https://playground-worker.pages.dev/api/prompt`) for embedding-based semantic search

*Gradio handling logic: lines 100-118 of [`agent/tools/docs_tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/docs_tools.py).*

## Fetching Full Documentation Pages with fetch_hf_docs

The `hf_docs_fetch_handler` provides the second critical capability: retrieving complete Markdown source for specific URLs identified during exploration. This handler:

1. Accepts a `url` parameter
2. Appends `.md` extension if missing
3. Authenticates using the session's HF token
4. Returns the raw Markdown content

*Implementation: lines 82-104 of [`agent/tools/docs_tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/docs_tools.py).*

## Practical Usage Examples

Below are minimal snippets demonstrating the tool workflow within an ML Intern agent context:

```python
import asyncio
from agent.tools.docs_tools import explore_hf_docs_handler, hf_docs_fetch_handler

class MockSession:
    hf_token = "hf_fake_token_123"   # In real use this is a valid token

async def demo():
    # 1️⃣ List the first 5 pages under the `transformers` endpoint that match "pipeline"

    args = {
        "endpoint": "transformers",
        "query": "pipeline",
        "max_results": 5,
    }
    text, ok = await explore_hf_docs_handler(args, session=MockSession())
    print("🗂️ Explore result:", ok)
    print(text)

    # 2️⃣ Grab the full markdown for a specific result URL

    page_url = "https://huggingface.co/docs/transformers/main_classes/pipelines"
    text, ok = await hf_docs_fetch_handler({"url": page_url}, session=MockSession())
    print("\n📄 Full page:", ok)
    print(text[:500])   # show a snippet

asyncio.run(demo())

```

**Typical output** includes structured listings with relevance scores (when querying) or paginated results (when browsing), followed by complete Markdown source when fetching specific pages.

## Summary

- **Dynamic endpoint discovery** leverages `COMPOSITE_ENDPOINTS` to group related documentation sets like `optimum` and `courses` into searchable units.
- **HTML scraping** via `_fetch_endpoint_docs` extracts navigation sidebars and retrieves raw `.md` files from `huggingface.co/docs/<endpoint>`.
- **In-memory indexing** uses Whoosh with `StemmingAnalyzer` to enable fast full-text search across titles and content without disk persistence.
- **Intelligent caching** via `_docs_cache` and `_index_cache` prevents redundant network requests and index rebuilds.
- **Dual-mode operation** supports both keyword-based relevance ranking (using `MultifieldParser`) and simple pagination for browsing.
- **Specialized Gradio handling** routes to dedicated endpoints for embedding-based search when accessing Gradio documentation.

## Frequently Asked Questions

### How does the search ranking work in ML Intern's documentation tools?

The system uses **Whoosh**, a Python search library, with a `MultifieldParser` that searches both the `title` and `content` fields of indexed documents. The schema employs `StemmingAnalyzer` to normalize terms, enabling matches between different word forms. Results include relevance scores calculated by Whoosh's scoring algorithm, with higher scores indicating stronger matches in the title or content fields.

### What is the difference between `explore_hf_docs` and `fetch_hf_docs`?

`explore_hf_docs` acts as a discovery engine that lists available documentation pages for a given endpoint, optionally ranking them by relevance when a query is provided. It returns titles, URLs, and preview glimpses. In contrast, `fetch_hf_docs` retrieves the complete raw Markdown source of a specific documentation page URL identified during exploration, enabling the agent to read full technical details.

### How does the system handle different Hugging Face documentation endpoints?

The implementation supports dynamic endpoint discovery through the `COMPOSITE_ENDPOINTS` mapping defined in [`agent/tools/docs_tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/docs_tools.py). When an agent requests a composite endpoint like `optimum`, the system automatically expands this to search multiple related endpoints (`optimum`, `optimum-habana`, etc.). For each endpoint, the tool scrapes the navigation sidebar of `huggingface.co/docs/<endpoint>` to build a complete index of available pages.

### Why is there special handling for Gradio documentation?

Gradio documentation receives special treatment because it utilizes a distinct architecture from standard Hugging Face docs. When the endpoint is `gradio`, the tool bypasses the Whoosh indexing pipeline and instead queries Gradio-specific endpoints: [`llms.txt`](https://github.com/huggingface/ml-intern/blob/main/llms.txt) for raw document listings and a playground worker API for embedding-based semantic search. This ensures the agent can access Gradio's specific documentation format and search capabilities.