# How the GitHub Enrichment Module Fetches and Classifies Repositories in interviewstreet/hiring-agent

> Discover how the GitHub enrichment module fetches and classifies repositories by extracting usernames, paginating the API, and using an LLM to identify top projects for evaluation in interviewstreet/hiring-agent.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: internals
- Published: 2026-07-14

---

**The GitHub enrichment module extracts a candidate's username from their profile URL, paginates through the GitHub REST API to retrieve all public repositories, classifies each project as either open-source or self-project based on license metadata, and uses an LLM to select the seven most impressive projects for evaluation.**

The interviewstreet/hiring-agent repository automates technical candidate screening by enriching resume data with public GitHub activity. At the heart of this pipeline lies the GitHub enrichment module, which transforms raw profile URLs into structured, classified project data that feeds directly into the scoring algorithm.

## Extracting and Sanitizing the GitHub Username

The enrichment process begins in [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) with the `extract_github_username` function (lines 116-131). This helper sanitizes the input URL by removing whitespace and applies regular expression patterns to extract the username from various GitHub URL formats.

```python
def extract_github_username(github_url: str) -> Optional[str]:
    if not github_url:
        return None
    github_url = github_url.replace(" ", "").strip()
    patterns = [r"https?://github\.com/([^/]+)", r"github\.com/([^/]+)"]
    for pat in patterns:
        m = re.search(pat, github_url)
        if m:
            return m.group(1)
    return None

```

The function returns `None` if no valid username is found, effectively halting the enrichment process for malformed URLs.

## Fetching Profile and Repository Data via the GitHub REST API

Once the username is isolated, the module implements a two-phase fetching strategy to minimize API calls and handle large portfolios. The `_fetch_github_api` helper (lines 29-33) executes the actual HTTP requests, while a disk-based caching layer (lines 19-25) stores responses in `cache/gh_githubcache_...` files to avoid redundant network traffic.

To retrieve the complete repository list, `fetch_all_github_repos` (lines 232-280) iterates over the paginated `/users/<username>/repos` endpoint, requesting 100 items per page until all public repositories are exhausted.

```python
def fetch_all_github_repos(github_url: str) -> List[Dict]:
    username = extract_github_username(github_url)
    if not username:
        return []
    repos = []
    page = 1
    while True:
        api = f"https://api.github.com/users/{username}/repos?per_page=100&page={page}"
        data = _fetch_github_api(api)
        if not data:
            break
        repos.extend(data)
        if len(data) < 100:
            break
        page += 1
    return repos

```

## Normalizing Metadata and Classifying Project Types

After retrieval, each raw repository object is normalized into a structured dictionary containing `name`, `description`, `html_url`, `stargazers_count`, `forks_count`, and `language`. The classification logic (lines 278-283) inspects the `license` field to determine the `project_type`: repositories with a non-empty license are flagged as `open_source`, while those without are categorized as `self_project`.

```python
def normalise_repo(raw: Dict) -> Dict:
    project_type = "open_source" if raw.get("license") else "self_project"
    return {
        "name": raw.get("name"),
        "description": raw.get("description"),
        "github_url": raw.get("html_url"),
        "github_details": {
            "stars": raw.get("stargazers_count", 0),
            "forks": raw.get("forks_count", 0),
            "language": raw.get("language"),
        },
        "project_type": project_type,
    }

```

The normalized list is then sorted by star count in descending order to surface the most popular work first. The module logs classification counts (lines 289-293) with status messages like `"📊 Project classification: Y open source, Z self projects"` to facilitate debugging.

## LLM-Driven Selection of Top Projects

Rather than passing all repositories to the scoring engine, the module uses `generate_projects_json` (lines 334-360) to transform the list into a compact JSON structure. This payload feeds into the `github_project_selection.jinja` template, which prompts an LLM to identify the seven most impressive projects based on popularity, complexity, and relevance.

The selection logic (lines 383-435) handles LLM response parsing, deduplicates any repeated selections, and implements a fallback mechanism that defaults to the first seven starred repositories if the LLM fails to return valid output.

## Integration with the Resume Scoring Pipeline

The `fetch_and_display_github_info` function (line 459) bundles the candidate's profile metadata, the full classified project list, and the LLM-selected subset into a single dictionary. This enriched payload is then consumed by [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) (lines 312-320), where it merges with traditional resume data before the final evaluation runs.

## Summary

- **Username extraction** uses regex patterns in `extract_github_username` (lines 116-131) to parse GitHub URLs and isolate the candidate handle.
- **API efficiency** is achieved through disk caching (`cache/gh_githubcache_...`) and paginated fetching via `fetch_all_github_repos` (lines 232-280).
- **Project classification** depends on the presence of a `license` field, separating repositories into `open_source` or `self_project` categories (lines 278-283).
- **Popularity ranking** sorts repositories by `stargazers_count` to prioritize the candidate's most visible work.
- **LLM curation** uses the `github_project_selection.jinja` template to select the top seven projects for evaluation (lines 383-435).
- **Pipeline integration** occurs in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) (lines 312-320), where GitHub enrichment data merges with resume content for final scoring.

## Frequently Asked Questions

### How does the module distinguish between open-source and self-project repositories?

The classification logic in `normalise_repo` (lines 278-283) checks the `license` field of each repository. If a license exists, the project is labeled `open_source`; otherwise, it is categorized as `self_project`. This binary classification allows the scoring pipeline to weight collaborative contributions differently from personal experiments.

### What caching mechanism prevents repeated GitHub API calls?

The module implements disk-based caching in `_fetch_github_api` (lines 19-33), storing API responses in files prefixed with `cache/gh_githubcache_`. This persistence layer ensures that subsequent runs for the same candidate retrieve data from local storage rather than hitting the GitHub API again, reducing latency and avoiding rate limits.

### How does the LLM select which projects to include in the final evaluation?

The `generate_projects_json` function (lines 334-360) prepares a structured payload of all normalized repositories. This data is passed to the `github_project_selection.jinja` template, which prompts the LLM to evaluate technical sophistication, community impact, and relevance. The selection logic (lines 383-435) then parses the LLM response, removes duplicates, and falls back to the top seven starred repositories if the LLM returns invalid or empty results.

### Where does the enriched GitHub data integrate into the hiring workflow?

After enrichment, the `fetch_and_display_github_info` function (line 459) compiles the profile and project data into a dictionary. This structure is imported into [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) (lines 312-320), where it merges with the candidate's resume information to inform the final technical assessment and scoring output.