# How GitHub Code Search in ML Intern Finds Examples Across Repositories

> Discover how ML Intern utilizes GitHub code search to efficiently locate and retrieve relevant code examples across repositories using fuzzy matching and ranked results.

- Repository: [Hugging Face/ml-intern](https://github.com/huggingface/ml-intern)
- Tags: how-to-guide
- Published: 2026-04-24

---

**ML Intern uses the `github_find_examples` tool in [`agent/tools/github_find_examples.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py) to scan repository file trees, score paths against curated example patterns using fuzzy token matching from the `thefuzz` library, and return ranked results that include ready-to-use payloads for the `github_read_file` companion tool.**

The **huggingface/ml-intern** repository provides an intelligent agent system that performs **GitHub code search** to discover reusable machine learning examples across public repositories. By recursively analyzing repository structures and applying fuzzy matching algorithms to file paths, ML Intern eliminates the need for manual documentation browsing. This article examines the deterministic eight-step pipeline implemented in the `github_find_examples` tool that enables cross-repository example discovery.

## The Eight-Step Discovery Pipeline

The `github_find_examples` tool implements a self-contained pipeline that transforms a repository identifier into a curated list of example files. This process is deterministic and requires only a single recursive API call to the GitHub tree endpoint.

### Step 1: Authenticate and Retrieve the Repository Tree

The process begins with authentication using the `GITHUB_TOKEN` environment variable. The tool calls the GitHub API endpoint `git/trees/<default-branch>?recursive=1` to retrieve the complete file tree of the target repository in one request. This is implemented in the `_get_repo_tree` function ([source L56-L108](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py#L56-L108)).

### Step 2: Filter for Blob Objects

After retrieving the tree, the system filters the response to retain only **blob** objects, which represent actual files rather than directories. For each file, the tool extracts the path, SHA, size, and URL to build a lightweight description object for downstream processing.

### Step 3: Score Against Example Patterns

Every file path receives a relevance score based on the **`EXAMPLE_PATTERNS`** constant defined at [source L15-L53](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py#L15-L53). This curated list includes patterns like `examples`, `scripts`, `tutorials`, and `notebooks`. The scoring uses fuzzy token-set matching from the **`thefuzz`** library via the `_score_against_example_patterns` function ([source L151-L158](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py#L151-L158)).

### Step 4: Apply Optional Keyword Filtering

If the user provides a keyword parameter, the tool calculates an additional fuzzy match score using `_score_against_keyword` ([source L160-L168](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py#L160-L168)). This allows the agent to find files related to specific concepts like "grpo" or "trainer" beyond just directory structure.

### Step 5: Calculate Three-Tier Priority Ranking

The `_get_pattern_priority` function ([source L171-L208](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py#L171-L208)) returns a priority tuple that determines sort order:

1. **Directory weight**: Files inside an `examples/` directory receive the highest priority.
2. **Pattern index**: The position of the matching pattern in `EXAMPLE_PATTERNS` (lower index equals higher priority).
3. **Path depth**: Shallower paths are preferred over deeply nested files.

### Step 6: Sort and Truncate Results

The tool sorts the candidate files by their composite scores and truncates the list according to the user-defined `max_results` and `min_score` parameters. This ensures the agent receives only high-confidence matches.

### Step 7: Handle Repository Errors with Similarity Search

If the target repository cannot be found, the `_handle_repo_tree_errors` function triggers `_search_similar_repos` ([source L112-L148](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py#L112-L148)) to look for similarly named repositories within the same organization. Rather than failing silently, the tool returns a helpful error message suggesting alternatives ([source L210-L264](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py#L210-L264)).

### Step 8: Format Output for Agent Consumption

The final output is a markdown-compatible string that includes a ready-to-use payload for the `github_read_file` tool. This payload contains the `repo` and `path` parameters, enabling seamless hand-off to the file retrieval stage.

## Fuzzy Matching and Scoring Implementation

The scoring system relies on the `thefuzz` library's token-set ratio to handle variations in path naming conventions. For example, a file in `demo_scripts/` matches against the `scripts` pattern with high confidence despite the prefix.

The priority tuple system ensures that traditional example directories take precedence over incidental matches. A file located at [`examples/grpo/train.py`](https://github.com/huggingface/ml-intern/blob/main/examples/grpo/train.py) will always rank higher than [`src/internal/examples_backup/train.py`](https://github.com/huggingface/ml-intern/blob/main/src/internal/examples_backup/train.py) due to the directory weight component in the three-tier logic.

## Integration with the Agent Architecture

### Tool Specification

The **`GITHUB_FIND_EXAMPLES_TOOL_SPEC`** constant ([source L404-L445](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py#L404-L445)) defines the tool's schema for the LLM-driven workflow. It specifies required parameters such as `repo` and optional parameters like `keyword`, `org`, `max_results`, and `min_score`, along with usage guidance for the agent.

### Registration and Routing

The tool is registered in [`agent/core/tools.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/tools.py) within the `TOOL_REGISTRY`. The handler function `github_find_examples_handler` ([source L448-L560](https://github.com/huggingface/ml-intern/blob/main/agent/tools/github_find_examples.py#L448-L560)) receives JSON arguments from the agent router, validates them, and forwards the request to the core `find_examples` function.

## Practical Code Examples

### Searching for Specific Keywords

You can invoke the search directly from Python to find example scripts containing specific terms:

```python
from agent.tools.github_find_examples import find_examples

# Search the "trl" repo for scripts containing "grpo"

result = find_examples(
    keyword="grpo",
    repo="trl",
    org="huggingface",
    max_results=5,
    min_score=80,
)

print(result["formatted"])

```

This call executes the full pipeline: retrieving the tree via `_get_repo_tree`, scoring against `EXAMPLE_PATTERNS` and the keyword, and returning the top five matches formatted in markdown.

### Invoking via the Tool Registry

When operating within the agent framework, the tool is accessed through the registry:

```python
from agent.core.tools import TOOL_REGISTRY

# The agent receives a tool request like:

#   {"name": "github_find_examples", "arguments": {"repo": "transformers", "keyword": "trainer"}}

response_text, success = TOOL_REGISTRY["github_find_examples"]["handler"](
    {"repo": "transformers", "keyword": "trainer"}
)

print(response_text)    # human-readable markdown

```

The handler manages parameter validation and error propagation automatically.

### Chaining Search with File Retrieval

A complete workflow combines example discovery with immediate content retrieval:

```python
from agent.tools.github_find_examples import find_examples
from agent.tools.github_read_file import read_file

# Step 1: Find candidates

search_result = find_examples(repo="trl", keyword="grpo", max_results=1)
file_payload = search_result["files"][0]  # Contains {'repo': 'huggingface/trl', 'path': 'examples/scripts/grpo.py'}

# Step 2: Read the file contents

content = read_file(**file_payload)
print(content[:500])   # preview first 500 characters

```

The `github_find_examples` output explicitly includes a copy-pasteable payload compatible with `github_read_file`, eliminating manual path construction.

## Summary

- **ML Intern** discovers examples using the **`github_find_examples`** tool, which performs recursive tree scans of target repositories via the GitHub API.
- **Fuzzy matching** against the **`EXAMPLE_PATTERNS`** constant using the `thefuzz` library identifies relevant scripts regardless of minor path variations.
- **Three-tier priority ranking** places files in official `examples/` directories highest, followed by pattern match specificity and path depth.
- **Defensive error handling** searches for similar repositories when the initial lookup fails, preventing silent failures.
- **Seamless integration** with `github_read_file` provides the agent with both discovery and retrieval capabilities in a single workflow.

## Frequently Asked Questions

### How does ML Intern authenticate with GitHub to search repositories?

The tool authenticates using the **`GITHUB_TOKEN`** environment variable. This token must be set in the environment before invoking any search operations, as the tool makes authenticated requests to the `git/trees` API endpoint to retrieve repository file structures.

### What determines whether a file is considered an "example" in the scoring system?

Files are scored against the **`EXAMPLE_PATTERNS`** constant, which includes directory names like `examples`, `scripts`, `tutorials`, and `notebooks`. The `_score_against_example_patterns` function uses fuzzy token-set matching to calculate relevance, meaning a file in `demo_scripts/` would match the `scripts` pattern with high confidence.

### Why are some example files ranked higher than others in the results?

The **`_get_pattern_priority`** function implements a three-tier ranking system: files located directly in `examples/` directories receive the highest weight, followed by the specificity of the pattern match (earlier patterns in `EXAMPLE_PATTERNS` rank higher), and finally the path depth (shallower paths are preferred). This ensures the most canonical examples appear first.

### Can the tool find examples if I mistype the repository name?

Yes. If the repository is not found, the **`_search_similar_repos`** function automatically searches for repositories with similar names within the same organization. The tool returns a helpful error message suggesting the closest matches rather than failing silently, allowing the agent to retry with the correct repository name.