# How the Hiring Agent Fetches GitHub Profile and Repository Information: A Technical Deep Dive

> Discover how the Hiring Agent fetches GitHub profile and repository data. Learn about its API wrapper, rate-limiting, caching, and LLM classification pipeline for project selection.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: deep-dive
- Published: 2026-07-09

---

**The Hiring Agent retrieves GitHub data by using a centralized API wrapper in [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) that handles authentication, rate-limiting, and caching, then passes structured data through Pydantic models to an LLM-driven classification pipeline in [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py) that selects the top seven projects.**

The interviewstreet/hiring-agent repository automates candidate evaluation by analyzing public GitHub data. Understanding how the system fetches GitHub profile and repository information reveals a sophisticated architecture that combines REST API integration, intelligent caching, and large language model processing to curate high-impact project portfolios.

## API Communication and Caching Layer

All GitHub interactions are centralized in **[`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py)**, which implements a resilient fetch layer that respects API constraints and eliminates redundant network calls.

### Deterministic Cache Key Generation

The module uses **`_create_cache_filename()`** to build deterministic cache file names for each API request. This function generates paths following the pattern `cache/gh_githubcache_<endpoint>[_params].json`, ensuring that identical requests map to the same filesystem location. When `DEVELOPMENT_MODE` is enabled, the system reads from these cache files instead of hitting the live API, allowing developers to replay data without consuming rate limits.

### Rate-Limit Handling and Authentication

The core **`_fetch_github_api()`** function manages all HTTPS communication with `api.github.com`. It implements several defensive mechanisms:

- **Authentication**: When the `GITHUB_TOKEN` environment variable is present, the function injects an `Authorization: token <GITHUB_TOKEN>` header to increase rate limits from 60 to 5,000 requests per hour.
- **Quota Monitoring**: The function parses `X-RateLimit-Remaining`, `X-RateLimit-Limit`, and `X-RateLimit-Reset` headers after every request. When fewer than ten requests remain, the system logs the deficit and sleeps until the quota resets.
- **Error Resilience**: All network calls are wrapped in `try/except` blocks. On failure, the function logs the error and returns `None`, allowing the evaluation pipeline to continue with partial data.

## Fetching Profile and Repository Data

Two public helper functions expose raw GitHub data to the rest of the system, handling URL parsing, API coordination, and Pydantic validation.

### Extracting User Profiles

The **`fetch_github_profile(github_url)`** function extracts the username from the provided URL, then calls `https://api.github.com/users/<username>`. Upon receiving a 200 response, it instantiates a **`GitHubProfile`** Pydantic model defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py), which validates fields such as `name`, `bio`, `followers`, `following`, and `public_repos`. If the API returns a non-200 status or the cache file is malformed, the function returns `None`.

### Retrieving Repository Metadata

The **`fetch_github_repositories(github_url)`** function hits `https://api.github.com/users/<username>/repos` and returns a list of repository JSON objects. For each repository, the module enriches the data by fetching contributor statistics from the `/contributors` endpoint, capturing `stargazers_count`, `forks_count`, and top-contributor usernames. This metadata provides the signal density required for downstream LLM classification.

## LLM-Driven Classification and Project Selection

Once raw JSON is available, the **[`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py)** pipeline orchestrates the conversion of repository data into a curated project shortlist.

The system injects GitHub metadata into the **`github_project_selection.jinja`** template located in `prompts/templates/`. This template instructs the LLM to select **exactly seven distinct, high-impact projects** and rank them based on criteria including stars, language diversity, recent activity, and relevance to the job description.

The LLM's JSON response is parsed by **`extract_json_from_response()`** in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py), which validates the structure before merging the selected projects back into the candidate's profile record. This structured output feeds directly into [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py), which computes the overall hiring score using the GitHub-derived insights.

## Integration Example

The following example demonstrates how to fetch GitHub data and process it through the classification pipeline:

```python
from github import fetch_github_profile, fetch_github_repositories
from transform import build_candidate_profile

# 1️⃣ Fetch raw profile and repository data

profile = fetch_github_profile("https://github.com/example_user")
repos = fetch_github_repositories("https://github.com/example_user")

# 2️⃣ Build the full candidate profile (includes LLM-driven project selection)

candidate = build_candidate_profile(
    basic_info={"name": "Example Candidate", "resume_text": "..."},
    github_profile=profile,
    github_repos=repos
)

# Access the curated project list

print(candidate["selected_projects"])

# → [{'name': 'awesome-app', 'stars': 540, 'language': 'Python'}, ...]  # exactly 7 entries

```

## Summary

- **Centralized API Wrapper**: The [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) module handles all GitHub communication through `_fetch_github_api()`, implementing robust rate-limit handling and token-based authentication.
- **Intelligent Caching**: `_create_cache_filename()` generates deterministic cache keys, while `DEVELOPMENT_MODE` enables offline development by reading from `cache/gh_githubcache_*.json` files.
- **Type-Safe Data Models**: `fetch_github_profile()` returns a validated `GitHubProfile` Pydantic model from [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py), ensuring data integrity before LLM processing.
- **Repository Enrichment**: `fetch_github_repositories()` aggregates standard repository metadata with contributor statistics to provide comprehensive project context.
- **LLM Classification**: The [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py) pipeline uses Jinja templates to prompt the LLM to select exactly seven high-impact projects, parsed via `extract_json_from_response()` in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py).

## Frequently Asked Questions

### How does the Hiring Agent handle GitHub API rate limits?

The `_fetch_github_api()` function monitors `X-RateLimit-Remaining` headers after every request. When the remaining count drops below ten, the system logs the threshold breach and sleeps until the `X-RateLimit-Reset` timestamp passes. Authenticating with a `GITHUB_TOKEN` increases the hourly quota from 60 to 5,000 requests.

### What caching mechanism does the Hiring Agent use for GitHub data?

The system uses `_create_cache_filename()` to generate deterministic filenames in the `cache/` directory following the pattern `gh_githubcache_<endpoint>[_params].json`. When `DEVELOPMENT_MODE` is active, the code checks for existing cache files before making network requests, allowing offline replay of API responses.

### How does the system validate GitHub API responses?

Raw API responses are validated through Pydantic models defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py). The `fetch_github_profile()` function attempts to instantiate a `GitHubProfile` object with the JSON payload, which enforces type checking on fields like `public_repos`, `followers`, and `bio`. Malformed or empty responses result in `None` being returned to the caller.

### Why does the LLM select exactly seven projects?

The `github_project_selection.jinja` template encodes a strict selection rule requiring exactly seven unique projects. This constraint ensures the output remains concise and actionable for hiring managers while forcing the LLM to prioritize the most significant repositories based on stars, recency, language diversity, and relevance to the target role.